• Home

Items filtered by date: December 2014

Friday, 26 January 2024 10:52

Panasonic JU-475-5 5.25 Floppy Disk drive

I have been looking for a 5.25" Floppy Disk Drive for recovering old disks. I wanted to make sure I got the right one and here i all of the notes on the drive I selected.

The Panasonic JU-475-5 will read all 5.25" disk formats so is a good choice and seems a popular choice.

  • General Information
    • Floppy disk - Computer History Wiki - Diskettes, or floppy disks, were a popular medium of storing information in the 1970s up until the 1990s.
    • Floppy Drive Connector - The floppy disk interface uses what is likely the strangest cable of all those in PCs today. It is similar to the standard IDE cable in that it is usually a flat, gray ribbon cable. It is unusual in terms of the number of connectors it has and how it is used to configure the setup of the floppy disks in the system.
    • The bitsavers Homepage
      • The PDF Document Format - This section tells you how the guy scans his documents.
  • Drive Information
  • Downloads
  • Mode/Speed Modifications and settings
    • DAVES OLD COMPUTERS - Modifying 1.2M drives for 300 rpm
      • Although 5.25" HD (1.2M) PC drives can read DS/DD media, they do so at 300kbps, which is non-standard for this media type, and has two disadvantages. (1) The disk images created will indicate a non-standard 300kbps data rate. and (2) it's hard enough to get a PC to read single-density - some that can do it at 250kbps cannot do so at 300kbps.
      • Fortunately, many 5.25" HD drives are fairly easy to modify to operate at 300rpm and 250kbps. This page describes how you can add a switch to a 5.25" HD drive to allow it to operate at either 300 or 360 rpm. The drive shown is a Panasonic JU-475 which is a very common HD drive used in PCs, however the techniques shown here can be applied to many other drives.
      • DAVES OLD COMPUTERS - Disk/Software Image Archive
    • Kaypro Hardware Part 2 | retrocmp.de
      • Panasonic, JU-475-5 AKO is mentioned
      • The special thing about the Panasonic and TEAC drives is that you can set the speed 360 or 300 RPM directly with a jumper. These are real dual-speed drives! Many other "dual" drives 1.2MB/360KB on the other hand rotate with constant 360 RPM and therefore write/read the data (for 360KB) with 300 kbit/s instead of usual with 250 kbit/s.
      • For this reason, you can use these regular "1.2MB" drives as true 720K/1M drives.
      • I forgot to mention that my Robie (now) has a Micro Cornucopia Pro-884 Max ROM. This can read and write all three Kaypro formats (200K, 400K and 800K) with a DS QD floppy disk drive (Panasonic JU-475-5: 2 sided, 96 tpi, 80 tracks).
  • Cleaning
    • Cleaning and lubricating a Panasonic JU-475 5.25" disk drive - YouTube | Poking Technology - Does some basic maintenance on a 1990-ish 5.25" floppy drive. I have two, both of which failed in exactly the same way at about the same time, and after some time trying to diagnose and fix them (including replacing some of the capacitors!), it's been suggested that all the really need is cleaning and relubricating.... so that's what I'm trying.
    • Floppy Drive Lubrication \n VOGONS
      • So I just got my hands on a pretty dirty 5 1/4 inch drive , I was just wondering if I should grease the movement rail and stepper rail with lithium grease? Or would something like silicon grease be better?
        • I hear vaseline is good for lubricating ones floppy.
        • r would something like silicon grease be better? = yes
      • Okay so I cleaned the read head and lubed the rails , the drives not reading disks or formatting ?
        Is it possible the drive is a dud?
      • SHE LIVES! , hokay so turns out the drive wasn't cleaned properly , there was some brown crud on the read head and it wasn't coming off easily , I didn't want to damage the heads so I went pretty light on it first time around , this time though having nothing to lose , decided to do my best to get the crud off , it took alot of cleaning , to the point I was sure id damage the drive , but after putting it back in my pc it suddenly detected the drive was drive A instead of B for some reason so I had to turn swap floppies off. After that I managed to format and then write 2 disks , so it seems to be working now.
  • Diagnostics
    • [SOLVED] Panasonic JU-475-4: issue with 5.25 floppy disk | VOGONS
      • When it tries to read a floppy, the head just moves back and forth a little and then stops, the "device not ready" message appears after that. The head movement is very short.
      • I cleaned the head with alcohol and lubricated everything. I can guarantee that the motor is working okay, I actually verified that by chance.
      • So first of all, these are 1-sided floppies. While most manufacturers made them all 2-sided by the time 5.25" drives got popular, and just branded as 1-sided, there is no guarantee they will work as such. Should though. And in any case these are double-density floppies and HD drives (such as your JU-475) can't write them very well. Sometimes it works, sometimes it doesn't, depends on the head and media (and contents). A HD drive should read a DD floppy though.
      • The extra hole would suggest these were used with something like C64 and flipped, if so you do need to reformat them. A PC FDC can't read GCR-encoded floppies. And if your drive can't properly write to DD media then these floppies will not be of much use as far as testing goes. But can still be used with 360k PC floppy drives.
      • To be sure if this drive works or not you need a HD floppy, and a known good one. So preferably not a NOS unformatted one. That being said, you mentioned a second floopy drive (3.5") that you have. Was it connected to the cable as well during testing? If not, connect it (make sure it has power connected as well) and try again.
      • As a rule of thumb if the floppy has an extra ring attached to the central hole, where the media is clamped to the spindle, it's a DD disk. Also that self-made write protect hole is a giveaway this particular floppy was used in 1-sided drive and flipped. So there is a good chance those are 1-sided floppies though yes, I can't be sure about that.
      • I just enabled floppy seek at boot, and now it can read the IBM DOS disks!
      • Formatted a C64 floppy as 360 kb one, first write was unsuccessful but tried again and it gone through. Read file back and was OK.

 

Published in Emulators
Read more...
Thursday, 09 November 2023 13:08

My Virtualmin Notes

This page will cover Webmin, Virtualmin and Usermin but I focus on getting a fully fledged Virtualmin server running on Ubuntu.

  • A lot of Webmin tutorials and information will apply to Virtualmin because Virtualmin is a plugin/module of Webmin.
  • HeadingsMap Firefox Add-On
    • This plugin shows the tree structure of the headings in a side bar.
    • It will make using this article as a reference document much easier.

Overview

General

  • Virtualmin UI overview 2021 - YouTube - This official video give you a great overview of the software.
  • Virtualmin is probably best when run with Apache.
  • Webmin vs. Virtualmin — RackNerd - Webmin and Virtualmin have been around for a long time, and they are known as one of the oldest free control panel's that still exist today.
  • Virtualmin doesn't support Mysql any more? - Virtualmin - Virtualmin Community - why MySQL is not longer supported and has been replaced with MariaDB.
  • Anyone using the new PHP version 8.3 on Virtualmin/Webmin? - General Discussion - Virtualmin Community
    • PHP doesn’t run “on Virtualmin” and Virtualmin does not use PHP for anything, so any questions about PHP should be about your apps and your OS and the repos you’re using. Virtualmin don’t care.
    • Virtualmin runs on Perl alone allow it to be completely separate from the actions on the server.
  • Virtualmin/Webmin is separate
    • Virtualmin/Webmin sits on top of Linux and only ever alters config files or issues commands, it does not change your Linux install beyond this so you can change things manually and Virtualmin will still work.
      • Some manual operations are frowned when Virtualmin is there to do these tasks for you to preserve consistency.
    • Virtualmin Framed Theme virtual-server-theme theme version 9.3 released - #12 by Joe - News - Virtualmin Community
      • No. Webmin runs under miniserv, a special purpose application server designed specifically for Webmin. The the only way to make something happen “before the theme” would be to make it so the theme can’t customize the login page and couldn’t customize any unauthenticated pages (of which there are several in Virtualmin, and removing those features would be pretty dramatic for many users), which isn’t really ideal, either.
      • Even when you run Apache or nginx in front of it, Webmin’s own web server is still running underneath; it’s possible to run Webmin directly under Apache, but it’d provide horrible performance, much weaker security (no 2 factor auth, no password timeouts, you’d have to configure any extra access controls in Apache, rather than in Webmin, etc.), and would not be themeable in a meaningful way (the application server transparently performs the path changes for themes). Running a proxy in front of Webmin might be a security win, but running Webmin directly under Apache, definitely, would not.
      • There are ways forward that may improve overall security on an architectural level, but they’re not simple, and we’re considering our options on those fronts. But, there is no magic bullet for security in a very large system.
    • Rebuild httpd.conf with all virtual hosts - Help! (Home for newbies) - Virtualmin Community
      • Unlike most other control panels, we don’t generate config files from templates, we edit them in place. We can’t possibly “generate” it because we always edit what is currently existent. The config file is the source of truth. This is a feature, not a bug.

Sites

Components Explained

  • Overview
    • Basic Questions on VirtualMin,WebMin,UserMin - #2 by Joe - Help! (Home for newbies) - Virtualmin Community - What is a Virtual Server? Is that a website/domain that is provisioned for hosting? In IIS this is a website. What do you use to set this up…Webmin or Virtualmin?
    • Whm = virtualmin and usermin =cpanel? - #3 by eugenevdm.host - Help! (Home for newbies) - Virtualmin Community
      • Usermin is a completely separate (optional) tool, unrelated to Virtualmin, though the Virtualmin installer installs Usermin because it is a webmail client, among other things, that integrates nicely with Virtualmin and Webmin.
      • Administrators and domain owners login to the same UI. When you login to Webmin (Virtualmin) as a domain owner user, that is not Usermin. Usermin is a webmail client, among other things, that runs on port 20000, by default.
      • The access domain owners have is configurable in Virtualmin, in Server Templates, Account Plans, and Virtualmin Configuration. You can grant Virtualmin domain owners a lot or a little access. It defaults to a little (though it could be even less, the default is intended to be a usable subset).
  • Coming from cPanel
    • Subdomains --> Subserver
    • cPanel: You can add unlimited subdomains into a cPanel account but they all share the same account resources and PHP settings.
    • Virtualmin:
      • Each domain (including subdomains) all have their own server instance called a `Virtual Server` and when not a parent virtual server, it is called a Subserver.
      • These Subservers can be owned by a single parent `Virtual Server`with which it will share various services with such as Mail and DNS records. PHP and other things are still separate thought.
      • By default all `Virtual Servers` are parents even if they do not have any child servers attached, and only Parent `Virtual Servers` have an account owner.
      • You can backup a parent and all of its subservers as one backup.
      • This format gives parity to cPanel accounts but with the added advantage of individual resources and settings for subdomains.
    • Virtualmin for cPanel Users – Virtualmin
      • This short guide will attempt to point out a few of the gotchas that we've found most commonly trip up former cPanel users trying out Virtualmin for the first time.
    • What are the key terminology differences between cPanel and Virtualmin | FAQ | Virtualmin — Open Source Web Hosting Control Panel
      • Virtualmin and cPanel have key differences in terminology — in Virtualmin, what cPanel refers to as “domain” is called “virtual server”, a “sub-domain” is called “sub-server” and “parked domain” is called “alias server”.
    • Understanding Virtual Server and Account Types | Virtualmin — Open Source Web Hosting Control Panel - Understanding the different virtual server and account types in Virtualmin is essential for effective web hosting and domain management.
  • Webmin
    • A Powerful and flexible web-based server management control panel.
    • This platform allows the installation of modules (plugins) to perform extra tasks.
    • Webmin is a web-based system administration tool for Unix-like servers, and services with about 1,000,000 yearly installations worldwide. Using it, it is possible to configure operating system internals, such as users, disk quotas, services or configuration files, as well as modify, and control open-source apps, such as BIND DNS Server, Apache HTTP Server, PHP, MySQL, and many more.
  • Virtualmin
    • This is a Webmin module.
    • Virtualmin users log into Webmin and they do not use Usermin for this purpose.
    • Virtualmin is available in two versions. Virtualmin GPL and Virtualmin Professional.
  • Usermin
    • This is a Webmin module.
    • This is another portal aimed towards techies and server admins, not Virtualmin users.
    • Usermin is a web-based interface for webmail, password changing, mail filters, fetchmail and much more. It is designed for use by regular non-root users on a Unix system, and limits them to tasks that they would be able to perform if logged in via SSH or at the console.
    • Most users of Usermin are sysadmins looking for a simple webmail interface to offer their customers. Unlike most other webmail solutions, it can be used to change passwords, read email with no additional servers installed (like IMAP or POP3), and setup users’ configurations for forwarding, spam filtering and autoreponders.
    • Usermin need to see logfiles (webserver) - Usermin - Virtualmin Community
      • Usermin is webmail plus a few other features you may want to enable. It is no for managing domains.
      • A Virtualmin user is for managing Virtualmin domains owned by that user.
      • I’ve seen people on the web suggest that Virtualmin==WHM, and Usermin==cPanel, but that’s simply wrong. Virtualmin is not split like that.
      • Usermin is not a management tool, it’s for end users to read their mail, manage mail filters and such, change their password, maybe use File Manager (for their own files, not websites), etc. You can grant them some extra privileges, but there is a user explicitly for what you’re trying to do and it is the Virtualmin user that was created when you created the domain.
    • All Virtualmin Virtual Server 'Owners' get a Usermin account created.
    • Very configurable.
  • Filemin
    • This use to be separate, but is now the integrated file manager of Webmin.
    • You can configure the File Manager not to lock users into their home
      • Webmin --> Webmin Users --> Permissions for all modules --> Root directory for file chooser
      • This would allow them to traverse upward through the directory tree to the logs for the domain.
      • This only works for system users that are not linked to Virtualmin and is not standard practice.

GPL vs Pro

Pro

  • Pro License | FAQ | Virtualmin — Open Source Web Hosting Control Panel
    • How do I upgrade from GPL to Pro?
    • How to upgrade Virtualmin license?
    • How do I renew an expired license?
    • How do I cancel a recurring license?
    • How do I upgrade or downgrade a license?
    • Where are my expired licenses?
    • Why do I see license error message?
    • How do I update payment information or find my invoices?
  • What counts toward your Domain count:
    • Each Virtual Server
    • Each Sub-Server
    • Each Sub-Domain (if you've enabled them)
    • Aliases do NOT count
  • Virtualmin Pro License subdomains - Virtualmin - Virtualmin Community
    • Sub-servers are full-featured domains that can have their own content, applications, mail, etc.(they can even have their own name unrelated to the parent domain). They count against the domain limit for this reason.
    • Aliases, which do not have their own content, do not count against the domain limit.
  • Create cPanel style subdomains (Manually) without increasing the domain count.
    1. How To Create Sub Domain In Virtualmin/Webmin - Petal Host - Most of the times we need to create sub domains in our accounts. Cpanel provides easy way to create sub domains. But when we talk about creating sub domain in Virtualmin/Webmin, it make us to think how to create sub domains as there is no direct option for creating sub domains.
    2. subdomains accounted for domains?? | Virtualmin
      • The following will not create a new Virtualmin 'Sub-Server' `Sub-Domain` account and will use Apache re-write to point the sub-domain to a sub-folder. This is a manual process.
      • Just wanted to point out that if you don't need all of the granular options/extended configurability that go with an actual server/sub-server account then a simple Apache rewrite rule can provide a simple "sub-domain" in terms of content presentation (mapping sub.domain.tld to a directory under said domain). Perhaps this is obvious but I hadn't seen any mention of it around.
      • I think spazzwig's suggestion is the best: having 'simple' sub-domains that just add a CNAME record to the DNS, add a <VirtualHost> directive in the Apache config file for that domain (or perhaps this can be done even without the VirtualHost entry, just using a redirect?), and then setup a folder inside domains/ (or perhaps subdomains/?).
      • I just tried it out manually, and all that's really needed is the CNAME, a new <VirtualHost> in the Apache domain conf file (or a new apache conf file would work just as well), and a subfolder inside the /home/domain.com. 
      • Or, you could write a "braindead sub-domains" module that creates a CNAME and a directive in Apache that points to a subdirectory in public_html. That'd be pretty trivial to write, and we wouldn't have any problem with folks doing that (we'd even answer any questions you might have about writing plugins)--but we aren't going to add more account types to Virtualmin.
      • It'd probably be cheaper to just write some code that'l simulate those subdomains, you can just map a subdomain to a folder with two or three lines.
        • DNS record, subdomain.domain.com. IN CNAME domain.com.
        • Apache ServerAlias subdomain.domain.com
        • Apache 
          RewriteEngine On
RewriteCond %{HTTP_HOST} ^subdomain\.domain\.com$
RewriteRule ^/(.*)$ http://domain\.com/subdomain
        • Now any request to subdomain.domain.com will load domain.com/subdomain.
        • Automating this is somewhat harder but I imagine fairly easy; I've not looked into it yet.
    3. Creating subdomains on the fly server template adjustment - Virtualmin - Virtualmin Community - Discusses and shows a user's methodology.
  • Upgrading to Pro
    • Pre-Sales Questions - Virtualmin - Virtualmin Community
      • Yes, upgrading usually just requires adding the serial number and license key in the “Upgrade to Pro” form. Virtualmin will switch software repositories to the Pro repos and upgrade your system to the Pro version.
  • Getting support when you have a Pro License
    • If you want to make a private support ticket, you send us a PM (done via the forum).
      • If you click the “Create Premium Support Ticket” button on the Support page it will open the forum with a private message window open and the recipient set to @staff. (This assumes you’re logged into virtualmin.com.)
      • Tagging us is not the same as sending a PM.
    • Support Module | Virtualmin — Open Source Web Hosting Control Panel
      • For Pro users there is a Virtualmin Support module that can be used effortlessly to submit a ticket.

        • Virtualmin --> System Settings --> Virtualmin Support
        • The support request is sent straight to a staff email inbox.
        • This support request will also include your system's information automatically.
        • This module will also all Virtualmin staff to login remotely to your server, if you give them permission.
  • Reset Pro Options visibility · Issue #797 · virtualmin/virtualmin-gpl · GitHub
    • The was raised by another user. during the use of GPL the user had disabled the showing of additional install scripts (Manage Web Apps) so was just left with the free ones. the user was keen to see what paid versions were available but did not want to go and re-install a whole virtualmin setup again just to see them.
    • Virtualmin --> System Settings --> Virtualmin Configuration --> Configuration category: User interface settings --> Show Pro features overview?
      • This option does not restore the hidden Pro script displays but hides menu items I think.
    •  Solutions: To show all hidden Pro features advertisements for root user.
      • Theme configuration --> Clear Cache
      • rm -f /var/webmin/modules/virtual-server/seenfeatures/root-pro-tips

Docker (via Cloudmin)

  • Setting Up Docker Virtualization – Virtualmin
    • Docker is not a true virtualization type like KVM, Xen or even LXC - instead it is a very lightweight container system that is typically used to run server processes in an environment that includes all their dependencies. Docker images normally contain a very basic Linux distribution and the files needed to run a single server like Apache, Nginx or MySQL.
  • Webmin Docker module - An easy way to deploy containers - Webmin Docker module allows to create and manage Docker containers easily. It involves setting up a Docker Host, adding docker images, etc.

Webmin / Virtualmin / Usermin

General

  • Virtualmin
    • allows the auto install of self signed SSL, you just have to find it.
  • Virtualmin Tab
    • Is always per single virtual server.
    • If there are more than Virtual Servers available in an account, they will be present in drop down menu. All are Virtual Servers are present in root or main admin account.
  • License Usage (GPL and Pro)
    • Am I allowed to rebrand Virtualmin, but retaining the MIT license? - Virtualmin - Virtualmin Community
      • Virtualmin is not under an MIT license.
        • There are some bundled components (e.g. JavaScript libraries, etc.) that are MIT and/or BSD-licensed,
        • Virtualmin GPL is licensed under the GPL and is subject to those terms,
        • Virtualmin Pro (everything in the pro subdirectory) is Copyrighted (and not freely distributable).
          • Pro subdirectory = /usr/share/webmin/virtual-server/pro/
        • Webmin is under a liberal BSD three-clause license (so the license notice needs to remain in place wherever it appears).
      • But, rebranding is fine for both GPL and Pro. You should be aware, however, that lang files get overwritten on updates, so you need to plan for that. The best way to handle that is…maybe a custom lang. You could also just make a patch using the diff command and then re-apply it after updates using patch. While it does change regularly, a patch will probably apply cleanly for quite some time. Or you could just script a search/replace using sed every time you update.
      • For the default page (the index.html that gets included in public_html when a new domain is created without content), assuming you’re replacing the whole thing, you don’t need to include any of our copyright notices. The web pages and apps you host on a Virtualmin system are not subject to our copyrights or licenses, unless they were made by us.
      • There are tools for adding logos and colors and stuff to the UI without needing to modify any files. It’s a configurable option. And, you can load custom stylesheets, which could more significantly alter appearance and insert logos or whatever.
    • Question about Virtualmin licensing for local development - Virtualmin - Virtualmin Community
  • How To Restore Deleted Module? - Webmin - Virtualmin Community
    • Q: I deleted fail2ban module because I want to use CSF, but now I want fail2ban back.
    • A: You could simply run apt-get install --reinstall webmin to restore any deleted files from the original package.
  • How to re-run the `Post-Installation Wizard`
    • Virtualmin --> System Settings --> Re-Run Install Wizard
  • Manage Virtual Server --> Switch To Server's Admin
    • This allows you to login as the owner of the currently selected virtual server.
    • cannot switch back to root after switch to server admin [#69822] | Virtualmin
      • Because for logging in, the given user name is used, there is no way to switch back from user account (server owner) to root (master administrator) account without compromising security, at least using our current model of authentication.
  • Webmin Modules menu
    • Virtualmin --> Webmin Modules
    • You only get this additional menu item when you login as a normal user (i.e. not root).
    • The available features are configured by permissions.
    • This menu can be turned on and off by the server admins on a per user basis.
      • Virtualmin --> Manage Virtual Server --> Edit Owner Limits --> Other restrictions --> Hide Webmin modules category in menu:
    • Uninstall Webmin Modules
      • You cannot uninstall a module.
      • They become active when the relative services is avaiable on the underlying OS, conversely if that serve is removed they will go back to an unused status.
      • Some unused modules might have a button to install the relevant service.
    • Deleting a module
      • Removing unwanted Modules for better security - Webmin - Virtualmin Community
        • We wouldn’t recommend removing (deleting) modules from Webmin installation directory but rather disabled those modules for users using:
        • Webmin --> Webmin Users --> Edit User --> Available Webmin Modules
      • Webmin --> Webmin Configuration --> Webmin Modules --> Delete
        • This should not be done for core Webmin Modules but this is for 3rd party plugins you might of installed and not longer want.
        • Some Modules might have their own delete routines.
        • Don't do this unless you really know why and what the consequences are.
  • Graceful Shutdown / ACPI Shutdown
    • Sending an "ACPI power down command" / "poweroff ACPI call" from either the Host OS, via a power button, or by running the `poweroff` command from within the Guest OS will cause the OS to shutdown gracefully.
    • Webmin on my Ubuntu OS is configured to use the poweroff and reboot commands which are part of the systemd and are the preferred commands.
    • poweroff sends an "ACPI power down command" to the OS which then performs a graceful shutdown.
    • Buttons
      • Webmin --> System --> Bootup and Shutdown -->
        • Reboot System
        • Shutdown System
    • Commands configured here
      • Webmin --> System --> Bootup and Shutdown --> Module config --> Configuration category: System configuration -->
        • Command to reboot the system
        • Command to shutdown the system

Install / Update / Upgrade / Uninstall

Some general information about this topic I have put together. These do not form my installation instructions, they are at the bottom of this article.

Tutorials

Install

  • If you have an administrative user with sudo ALL privileges, commonly the first user on an Ubuntu system, you can use that user to login to Virtualmin.
  • Downloading and Installing Virtualmin – Virtualmin | Official method)
  • Virtualmin Professional – Virtualmin - Virtualmin GPL is already an extremely powerful and flexible virtual hosting control panel, so we're frequently asked about the differences between Virtualmin GPL and Virtualmin Professional. So, if you were wondering whether you should upgrade, you've come to the right place.
  • Once you hve downloaded the install script you can run the script with a help switch`install.sh --help` and it will give you help information and will not install anyting or modify your system.
  • Automated Virtualmin Installation – Virtualmin
    • There are two methods for installing Virtualmin. The first is a fully automated script described in this document, and the other is a manual installation documented in the Manual Virtualmin Installation page.
    • This is the recommended method.
  • Manual Installation – Virtualmin
    • Unlike the Automated Virtualmin Installation, to make use of this installation type, your OS does not need to be freshly installed, nor does it need to be a supported operating system.
    • This method, however, requires significantly more knowledge on the part of the person doing the installation, and a much larger time investment to insure that all necessary configuration is performed and all Virtualmin managed services are working correctly.
  • The install will appear to get stuck but it is just slowly downloading assets. In total the installation took about 20mins.
  • The MySQL module installs MariaDB

Update / Upgrade

GPL <--> Pro

  • Changing license will change the repositories used.
  • GPL and Pro are interchangeable in the sense of when the license expires, nothing will break but functionality will be reduced?
  • Uninstalling Virtualmin | Virtualmin — Open Source Web Hosting Control Panel
    • If you no longer need the features of Virtualmin Professional, but wish to continue to use Virtualmin on your system, you can downgrade quite easily by running: 
      virtualmin downgrade-license --perform
    • It will completely replace Virtualmin Pro package with GPL variant, making it impossible to use Pro features anymore. It will also disable all reseller accounts. By downgrading to GPL, you will no longer support the product development.
  • What happens when updating from GPL to Pro to a server with ~30 domains? Pro has limit to 10 domains. - Virtualmin - Virtualmin Community
    • If you have 30 domain on a 10 domain server – two things would happen:
      1. You’d see an warning each time that you logged in as the Master Admin that you had exceeded the domain name limit.
      2. You wouldn’t be allowed to add any additional domains until you were under the 10 domain limit again.
  • When the Pro license expires, will the /pro/ folder be deleted?

Uninstall

  • Uninstalling Virtualmin – Virtualmin
    • There are many levels of uninstalling Virtualmin.
    • --uninstall - This should never be done on a system that is in production. It is very destructive. It is primarily for use when you tried an installation option (for example using Nginx instead of Apache) and have decided to change after trying it out.
    • Uninstalling / Downgrading Virtualmin Professional to GPL, both are covered here.
    • The 'virtualmin change-license' command is used for license changes and renewals. Check the license FAQ for details.
  • How can I uninstall Webmin? - FAQs | Webmin
    • Just run the command /etc/webmin/uninstall.sh. If you have installed the rpm package of Webmin, you can also use rpm -e webmin, or dpkg -r webmin if you have installed the deb package, or if you have installed the Solaris package you can use pkgrm WSwebmin command.

Custom Menu Links

Add additional items into the Virtualmin dashboard menu.

GPL

Basic and theme based, but will do the job for most

  • Theme Configuration --> Configuration category: Navigation menu --> Extra entries
    • Read the Tooltip.
    • Using Authentic theme you can add extra links at the bottom of the navigation menu in the dashboard.
    • The injection done by the theme level so all injections are on a global level
    • The links can be configured for display to 5 pre-set user groups.
    • The example code from the tooltip but easier to read, but currently the field will not except this format and needs to be flattened.
      {
    "extra": [{
        "title": "Google Mail",
        "link": "https://gmail.com/",
        "icon": "google",
        "level": "0,1,2,3,4"
    }, {
        "title": "BIND DNS Server",
        "link": "/bind8",
        "icon": "server"
    }, {
        "title": "Usermin",
        "link": "/",
        "icon": "envelope",
        "port": "20000",
        "target": "_blank"
    }]
}
  • The Icons
    • There is a limited set of icons you can choose from.
    • The icons are a custom set of FontAwesome icons and you can preview most of them here: Authentic Kit Demo
    • Take fa- off and you have your icon name to sue in the code above
    • For reference, the icons are base64 encoded and in `bundle.min.css` with the font name `Authentic`.
    • authentic-theme/unauthenticated/css/bundle.min.css 
      /*
 * Authentic Theme (https://github.com/authentic-theme/authentic-theme)
 * Copyright Ilia Rostovtsev <ilia@virtualmin.com>
 * Licensed under MIT (https://github.com/authentic-theme/authentic-theme/blob/master/LICENSE)
 */
body{text-rendering:optimizeLegibility}@font-face{font-family:Authentic;src:url(data:application/font-woff2;charset=utf-8;base64,d09GMg............ 
    • Simple GUI for Virtualmin - #21 by Joe - Virtualmin - Virtualmin Community
      • Current Webmin and Virtualmin icons are from Nuvola from about the same era as Oxygen (i.e. also probably about 15 years old, and not really actively maintained or growing). I’d contracted David Vignoni, the creator of Nuvola, to do several dozen additional Webmin and Virtualmin specific icons back then. The problem with any icon set is that even quite large icon sets will need dozens or hundreds of additional custom icons. That’s surprisingly expensive.
      • We also heavily use icons from Font Awesome (the Open Source collection, we can’t use the Pro icons as they aren’t redistributable).

Pro

File Manager

Terminal

  • Terminal | Webmin
    • About The Terminal module in Webmin is a feature that allows you to access and interact with the command-line shell of your server or system directly from within the Webmin interface.
  • Starting with Webmin 2.200, all sudo-capable users will log in as themselves instead of as root.
    • To disable this limitation, as root:
      • Webmin --> Webmin Users --> root: Edit Webmin User / Terminal: Module Access Control page, and set the Enforce sudo-only privileges option to No.
      • Webmin --> Webmin Users --> Edit Webmin User --> Available Webmin modules --> Terminal (: Module Access Control) --> Enforce sudo-only privileges: --> No
    • Reason for this
      • Joe: It’s just making terminal behavior align with what all other terminal sessions would do, when logged in a non-root sudo-capable user.
      • Also, it is bad practice to use the root account for regular admin operations and so this was changed to follow that paradigm.
    • Feedback from a first time user - Blue Skies - Virtualmin Community
      • ilia
        • If you want to be root, then just log in as root.
      • Joe
        • Webmin treats all sudo ALL capable users as root (which was to accommodate systems that don’t have a root user password set…Ubuntu started doing that by default a decade or so ago, leading the charge on that), but that’s not what users new to Webmin expect.
        • I think people should still be aware, however, that a sudo ALL user in Webmin has the ability to use all the modules and such with no limits, by default; that’s expected/intended (after all if you have sudo ALL privileges, you can sudo su - to become root or sudo to change anything on the system. This isn’t “privilege escalation” in the security exploit sense…a user with sudo ALL is already root-capable.
        • And, no, this won’t effect anything else in Webmin/Virtualmin. It’s just making terminal behavior align with what all other terminal sessions would do, when logged in a non-root sudo-capable user.
      • anon50555658
        • I honesly hardly remember when I was user root on any system last time. 99.99% of all commands demanding root is runnable using sudo, no need to ever switch to root.
  • Webmin --> Tools --> Terminal
    • This will use the credentials of the logged in user.
  • Virtualmin --> Terminal
    • This will use the owner's credentials of the select Virtual Server.
  • 'virtualmin help' command locks the terminal
    • It does not lock the terminal, rather you should use one of these commands to quit: 
      q or :q

SSH

Ports being used

  • Webmin, Virtualmin, Usermin and other service Ports?
    • you can see them all here
      • Webmin --> Networking --> FirewallD:
    • What are the unamed ones for
      • 20: FTP Passive Mode Data
      • 22: SSH/SFTP
      • 2222: SFTP (FTP over SSH) (this use ProFTPd jail features and doesn’t need configuration?).
      • 10000: Webmin
      • 10000-10100: Webmin RPC?
      • 20000: Usermin
      • 49152-65535: PASV (used for FTP Active mode and other things)
    • What ports should be opened for Virtualmin in firewall? – Server Administration – vpsfix.com Forum - This is a question people ask when configuring firewall for Virtualmin. This is really important on platforms like Amazon Web Services and Google Cloud platform because they have a built-in firewall blocking all connections.
    • Acronyms:
      • SFTP = SSH FTP
        FTPS = FTP-SSL
  • Change Virtualmin/Webmin port
  • Change Usermin port
    • Webmin --> Usermin Configuration --> Ports and Addresses

Webmin (only)

  • Restrictions / Security
    • Restrict access to Webmin by IP or Hostnames.
      • Webmin --> Webmin Configuration --> IP Access Control --> Allowed IP addresses:
    • Restrict access to Webmin of a single user by IP or Hostnames. In this case the root account.
      • Webmin --> Webmin Users --> root account --> Security and limits options --> IP access control --> Only allow from listed addresses: might accept range
    • Webmin can also be configured to allow users who have sudo privileges for all actions to login as a root-level user.
      • Webmin --> Webmin Users --> Configure Unix User Authentication --> Allow users who can run all commands via sudo to login as root
      • How can I connect to webmin as a sudo user - Super User
        • Now you should be able to login as any user that has ALL sudo privileges. This feature was added to accommodate systems like Ubuntu that do away with having a "root" account, by default (Ubuntu has a root account, but it has no password and the first user created gets added to the sudoers file automatically).
        • This option is enabled, by default, on systems that we know meet this description (like recent Ubuntu releases), I think.
  • Authentication
    • Authentication - Webmin Configuration | Webmin
    • For HTTP authentication, there is no session tracking at all - the browser sends the username and password for every request!
    • Clear login sessions
      • Webmin --> Webmin Users --> View Login Sessions

Usermin (only)

  • General
    • All Virtualmin Virtual Server 'Owners' get a Usermin account created.
  • Restrictions / Security
    • Restrict user's Usermin permissions
      • Webmin --> Usermin Configuration --> Module Restrictions --> Add a new user or group restriction
    • Restrict access to Usermin by IP or Hostnames.
      • Webmin --> Usermin Configuration --> IP Access Control --> Allowed IP addresses:

Themes

  • General
    • Creating Overlay Themes | Webmin - This page explains how to create overlay themes, which are a new feature in Webmin 1.450 and later. These allow you to easily modify the colours, icons and CSS of another theme, without having to create or duplicate its entire layout.
    • Authentic Theme allows you to add your own JavaScript, JQuery, CSS and Perl to override things you don't like.
    • The services shown on the `Server Status` accordian panel is hardcoded in Virtualmin.
  • Restore hidden dashboard accordian panels
    • On the dashboard you have many panels and you can disable them but to re-enable them go here:
    • Theme Configuration -->  Configuration category: Dashboard and real-time monitoring --> Hidden accordians
    • This currently will only appear when you have any panels hidden.
  • Unhide Widget
    • In the `Content layout` widget there is an option called 'Hide This Widget' which will hide the widget from the user permanently.




    • once hidden you can re-enable it in the Theme Configuration:
      • Theme Configuration --> Configuration category: Table display --> Show content layout control: Yes
  • How do I change between table and list layout?
    • Click on the `Content layout` icon at the top left of the content area. It looks like a 3x3 table icon.
      • NB: it is not visible on the dashboard, choose a page with some regular content and the widget will appear. It is present on the Theme configuration page.
    • Choose your preferred layout variation:
      • Vertical (Table Style / Double Column)
      • Horizontal (List Style / Single Column)
    • Click Save
  • Change Theme

Email

General

  • Where are these default emails from? 
    abuse@example.com
postmaster@example.com
hostmaster@example.com
webmaster@example.com
    • These emails addresses are usually created by an unmodified 'Default Settings' Server Template, but can be created from any Server Template.
      • Virtualmin --> System Settings --> Server Templates --> 'Default Settings' --> Mail for domain 
    • These emails are aliases of the 'Primary email address'.
      • Virtualmin --> Edit Users --> 'Virtual Server Owner' --> Email Settings --> Additional email addresses
    • To completely disable any email on the primary account
      • Virtualmin --> Edit Users --> <username> --> Email Settings --> Primary email address enabled: No
    • Virtualmin --> Edit Users --> <username> --> Email Settings --> Additional email addresses: <remove any found here>
  • Enable Email Forwarding
    • Email Mail Alias
      • Virtualmin --> Edit Mail Aliases --> Add an alias to this domain
      • This allows you to create a forwarder without requiring a real mailbox.
      • This feature will also allow you to create delivery groups.
    • How to setup email forwarding – Virtualmin - This tutorial will cover how to setup email forwarding for a user from within Virtualmin. Only works for an account that already exists.
    • Email forwarders based on conditions - #4 by maycobb - General Discussion - Virtualmin Community - Yes, it’s possible to set up email forwarding based on conditions such as the sender and subject line in Virtualmin. Virtualmin is a web hosting control panel that includes features for managing mail servers, among other things. Here’s a general guide on how you might achieve this in Virtualmin.
  • Commandline
    • virtualmin modify-spam
      • Change spam filtering and delivery settings for a virtual server
      • To enable the spamtrap and hamtrap aliases for the selected virtual servers, you can use the "--spamtrap" command-line flag. Similarly, to remove them use the "--no-spamtrap" flag. When enabled, users will be able to forward spam to spamtrap@theirdomain.com for adding to the domain's blacklist.
    • virtualmin set-spam
      • Change the spam and virus scanners for all domains
  • Misc

Email and Anti SPAM Technologies (Email)

The various technologies to prevent SPAM.

  • SPF
  • DKIM
  • DMARC
    • DMARC is a record that explains what verification methods are available from among those other email technologies and tells receivers they should reject any mail that claims to be from your domain that doesn’t meet those requirements
    • How do I change the DMARC emails addresses?
      • Change the email address in the related server template
        • Virtualmin --> System Settings --> the template --> Edit template section: DNS domain --> (Reporting URI for forensic reports | Reporting URI for aggregate reports)
      • Regenerate the DNS records
        • Virtualmin --> DNS Settings --> DNS options
        • Click Save
    • How to set a good defaults for DMARC records on a Virtualmin server with many domains » Vander Host Knowledgebase
      • If you’ve got a perfect Virtualmin setup, you might notice that MX Toolboxes’s Email Health checker complains about DMARC.
      • Next you might wonder what DMARC setting will not only work for your own domain, but the other 5000 domains under your management.
  • DANE
  • TLSA
    • A TLSA record which is basically a hash from the received certificate.
    • TLSA Record Generator · SSL-Tools - Use this generator to create a TLSA entry as described in RFC 6698 for your domain. TLSA entries are required by DANE (DNS-Based Authentication of Named Entities).
  • MTA-STS
    This technology is not currently added to Virtualmin but can be added manually if you want it.
    • MTA-MTS prevents email being sent to servers in paintext by checking with the remote server that it supports encryption, if it doesnt your email server will not send your email to it.
    • What is MTA-STS? How to setup a MTA-STS Record - MxToolbox
      • This has instructions on how to setup MTA-STS on Virtualmin.
      • MTA-STS, which stands for Mail Transfer Agent Strict Transport Security, is an email standard that secures inbound email and prevents attackers from exploiting a weakness in standard SMTP security.
      • The MTA-STS standard, at its core, is a combination of having all of your email servers using Transport Layer Security (TLS), having valid publicly-trusted certificates for those servers, a published DNS record, and a TXT file.
      • MTA-STS, once implemented, actively enhances security of inbound email to your domain from attackers looking to intercept unsecured emails.
      • MTA-STS Lookup - Check domains for Inbound Transport Layer Security (TLS) Enforcement - MxToolbox
    • Introducing MTA-STS for Exchange Online - Microsoft Community Hub - The SMTP protocol isn’t secure and wasn’t designed to be. Email sent in the early days of the Internet were the digital equivalent of sending a postcard through the postal system. Eventually, Transport Layer Security (TLS) encryption was added to protect SMTP communications. But to maintain backward compatibility, it was never made compulsory and even today it’s used only opportunistically by senders.
    • Using the Mail Transfer Agent Strict Transport Security (MTA-STS) protocol in your organisation - GOV.UK
      • Mail Transfer Agent Strict Transport Security (MTA-STS) is a protocol which tells services that are sending your organisation email that your domain supports Transport Layer Security (TLS) 1.2 or higher. This protocol makes email less vulnerable to middleperson attacks and allows the receiving email service to enforce encryption, without the risk of delivery failing.
      • If the sending email service does not support MTA-STS or TLS, the email could still be delivered unencrypted.
      • As an email administrator, you do not need your provider to support MTA-STS to protect emails sent to your organisation. The sender of the email has to support MTA-STS on outbound email for the protocol to work.
    • How To Configure MTA-STS and TLS Reporting for Your Domain Using Apache on Ubuntu 18.04 | DigitalOcean - In this tutorial, you will learn how to configure MTA-STS and TLSRPT for your domain name, and then interpret your first TLS Report. While this tutorial covers the steps for using Apache on Ubuntu 18.04 with a Let’s Encrypt certificate, the MTA-STS/TLSRPT configuration will also work on alternatives, such as Nginx on Debian.
    • 2. Create an MTA-STS policy - Google Workspace Admin Help - Set up MTA-STS for your domains by creating and publishing a policy for each domain. The policy defines the mail servers in the domain that use MTA-STS. Each domain must have a separate policy file.
    • A webinar record for MTS-STS by Synametrics Technologies. - YouTube | Synametrics Technologies
      • This is a recording of a webinar that occurred on Aug 22, 2023. It talks about, what is MTA-STS, why you need to enable it for you domain, how to publish the policy files, update DNS records and view a summarized version of TLS-Reports.
      • The first bit is very useful but then it is very specific to the Xeams platform.
      • Requires (per domain):
        • Two DNS Records
          • _mta-sts.example.com
          • _smtp._tls.example.com
        • One webpage
          • https://mta-sts.example.com/.well-known/mta-sts.txt
          • A web server to serve this file (Virtualmin might llow you to use an alias here instead of another virtual server)
          • Host name must match
          • configure DNS server for mta-sts.example.com to point to thios web serverUse a trusted SSL cert
          • Must be opn port 443
      • Example mta-sts.txt file
    • What is MTA-STS ? (2024) - YouTube | PowerDMARC
      • The MTA-STS protocol specifies to an SMTP sending server that emails addressed to your domain must be sent over a TLS-encrypted connection. In case an encrypted channel cannot be negotiated, the email is not delivered at all, instead of being delivered as cleartext.
      • MTA-STS prevents:
        • DNS spoofing attacks
        • SMTP downgrade / MITM attacks
    • add MTA-STS support · Issue #808 · virtualmin/virtualmin-gpl · GitHub - excellent description and discussion of MTA-STS.
    • Stronger Email Security with SMTP MTA STS: Strict Transport Security - An indepth article.
  • ARC
    • ARC Specification for Email
      • The Authenticated Received Chain, or ARC, has been published by the IETF as RFC 8617. The specification is available as an HTML, PDF, and plain text document.
      • What is ARC?
        • When an email sender or Internet domain owner uses email authentication to make it easier to detect fraudsters sending messages that impersonate their domain, some services like mailing lists or account forwarding may cause legitimate messages to not pass those mechanisms, and such messages might not be delivered. These services may be referred to as intermediaries because they receive a message, potentially make some changes to it, and then send it on to one or more other destinations. This kind of email traffic may be referred to as an indirect mailflow.
        • ARC preserves email authentication results across subsequent intermediaries (“hops”) that may modify the message, and thus would cause email authentication measures to fail to verify when that message reaches its final destination. But if an ARC chain were present and validated, a receiver who would otherwise discard the messages might choose to evaluate the ARC results and make an exception, allowing legitimate messages from these indirect mailflows to be delivered.
  • TLS Reporting
    • to be added if not just part of MTS-STS
    • What is TLS Reporting TLS-RPT (2024)? - YouTube
      • Understand in simple terms what is SMTP TLS Reporting (TLS RPT).
      • SMTP TLS Reporting (TLS-RPT) is a standard that enables the reporting of issues in TLS connectivity that is experienced by applications that send emails and detect misconfigurations. It enables the reporting of email delivery issues that take place when an email isn’t encrypted with TLS. In September 2018 the standard was first documented in RFC 8460.
  • BIMI
    • cc

Troubleshooting (Email Diagnostics)

  • Emails going in SPAM
    • Email going to spam. Dmarc, dkim, spf settings to improve delivery rate? [#68798] | Virtualmin
      • Webmin --> Servers --> Bind DNS Server --> Choose domain --> Choose type DMARC and finally i created the record adding 100 to percentage of messages to apply policy
      • Virtualmin --> Email Settings --> DomainKeys Identified mail and save.
      • If this is a proper way to create a DMARC record. = Not exactly. You should rather go to: Virtualmin --> DNS Settings --> DNS Options --> DMARC record enabled
      • Should i do it manually for every virtual server (DMARC)? = For DMARC records, you would have to edit Server Templates and its BIND DNS Domain and enable Add DMARC DNS record.
      • What else i can do in order to improve mail deliverability? = I would set DMARC policy to "reject". SPF record should still and also be enabled on mentioned DNS Options page above.
      • here is no a global option in order DMARC is enabled by default after a new virtual server created or migrated right = However, in case you have hundreds of domains and doing it manually is difficult, you could use Virtualmin CLI to run mass update: 
        virtualmin modify-dns --domain name | --all-domains | --all-nonvirt-domains
    • All Mails are going to spam in gmail - Virtualmin - Virtualmin Community
      • A worked example with solutions
    • Verify your DKIM, SPF, DMARC (optional) and other email technologies are enabled and configured correctly. Having a technology running but misconfigured is just can be as bad as being recognised as a spammer.
    • SpamAssassin Configuration Tips - #8 by jimr1 - Virtualmin - Virtualmin Community 
      ## Look for messages from spamd
journalctl -t spamd --since "1 hour ago"

## Ensure spamd is running
systemctl status spamassassin

## Additional
you may need/want to remove the --since "1 hour ago" if you get no results from journalctl
  • I’m able to receive email, But Can’t Send it
    This is not the same as your emails getting classified as SPAM
    • Cause
      • Your VPS or ISP is blocking outbound traffic on port 25 while allowing incoming traffic on port 25.
      • This is extremely common and is usually the default.
      • This is used as a SPAM prevention method.
    • Verify port 25 is blocked for outgoing traffic
      • Try telnet on port 25 from your server to some other mail server, like Google. That tests outbound, and if blocked, it will time out. 
        telnet smtp.google.com 25
    • Solution
      • Contact your VPS or ISP and ask them to unblock port 25.
      • Many providers will unblock port 25 if requested, and if you don’t use it to send spam, but not all will allow this.
    • Workaround
      • Use a SMTP mail relay service such as mailgun or sendgrid.
      • The alternative is using a mail relay service like Mailgun, Sendgrid, Amazon SES, etc. Many have a free tier that would be sufficient for very light usage, and SES would only cost a few cents a month if you’re just using it for normal mail, and not bulk mail.
    • Notes
      • You can only send email using port 25.
      • You can’t make the rest of the world accept mail on any other port, you either have port 25 or you can’t send mail directly from your server to other servers on the internet.
      • Check your logs. and postfix mail que for more information.
  • Who is SPAMMING?
    • Who is sending emails
      • My domaine or server cant send to gmail - Virtualmin - Virtualmin Community
        • Look at the Postfix mail queue. Since you can’t send any mail, you probably have a lot of mail in the queue. Are they legitimate? Are they spam?
          • Webmin --> Servers --> Postfix Mail Server --> Mail Queue
        • If you see spam in the queue:
          • You can see who it’s from.
          • Edit a message and see what account is sending the email.
          • You can then drill down into the mail log or journal for the postfix unit to find out more details about how that message ended up in the queue.
          • If it’s a domain owner user (instead of a user within a domain) you can probably assume it’s from an exploited web application dropping it into the queue locally, and you’ll then know which domain is hosting an exploited web app.
        • If you do not see spam in the queue:
          • This means they’re sending without Postfix.
          • You then need to check the access logs for a site that’s getting a lot of requests to a script that you don’t recognize. Maybe it’ll show up in the error log.
          • I’ve also told you how to check for outgoing packets on port 25, and how to find out what process and user is making the connection.
        • Additional
          • Check for outgoing packets on port 25, and how to find out what process and user is making the connection.
          • You need to spend some time reading logs so you can figure out if what you’re seeing is abuse, and which user or application is involved.
          • Apply mail rate limits
            • If you apply mail rate limits (not per minute or per hour, but per day - say 100 per day) the virtual server which sends spam will be blocked and the virtual server which does not will be able to operate normally.
            • Only if they’re using Postfix to send mail. If they’re abusing a web application to send using their own MTA implementation, it will not go through Postfix, and any limiting you do in Postfix will do nothing.
            • You should have mail rate limits on anyway to prevent massive abuse of your server.
      • Can I find out who is broadcasting emails on my server? [#14192] | Virtualmin
        • Someone could be sending a lot of emails, but a common cause of what you're seeing occurs when a bot breaks into a website, and uses it to send email. If that's the case, you'd likely see a lot of emails from one particular user in the mail queue.
        • You can also use the bandwidth monitoring to see if one domain in particular is sending a lot of emails (in System Settings -> Bandwidth Monitoring).
        • Often, looking in the email queue or bandwidth usage makes it obvious who or what is at fault.
      • Protection against spam - #27 by ID10T - Virtualmin - Virtualmin Community
        • This might be useful. It uses the destination port and that’s what we are looking for. Not what is coming in on port 25 but what is going out.: 
          watch -n .1 ss -te 'dport == :25'
        • the -n .1 us the time basis for which watch renews. You can lengthen this time to try and get more stable output.
    • Logs
      • Webmin/Virtualmin
        • Webmin --> Systems --> System Logs
        • Webmin --> Systems --> System Logs Viewer --> File /var/log/mail.log
        • Virtualmin --> Logs and Reports --> Search Mail Logs (Pro Only)
        • Virtualmin --> Logs and Reports --> Bandwidth Graph
      • Postifx

Virtualmin Install Scripts (Manage Web Apps) (3rd Party Apps)

  • Installing by script (eg phpMyAdmin, RoundCube)
    • You are able to to install from a selection of apps using inbuilt scripts. The Pro version has many more.
      • Virtualmin --> Manage Web Apps
    • The free version includes all of the major ones you need.
    • Installable Applications – Virtualmin
  • Failed to install script : This script cannot be installed (phpMyAdmin)
     
    • Fatal Error!
Failed to install script : This script cannot be installed, as this virtual server does not meet its requirements : phpMyAdmin requires a MySQL database
    • This is most likely caused by:
      • The MariaDB service not being enabled for this account.
      • No available database to install into.
  • Get rid of AWStats from public directory / There are symlinks and an icon folder (icon, awstats-icon, awstatsicons)
    • If you disable AWStats for a Virtual Server, then the icon folder and the Symlinks will disappear for that Virtual Server.
    • keep "stats" and other icon folders outside of the public_html - Virtualmin - Virtualmin Community
      • Q: Virtualmin creates these default folders: awstats-icon, awstatsicons, icon stats. I’d prefer to keep stats and other icon folders outside of the public_html folder.
      • A:
        • These files and symbolic links are used by “AWStats” so if you don’t need this feature you simply can remove the feature from the domain and safely remove folders/symbolic links.
        • That’d be somewhat tricky. You’d need an additional Directory section added to each VirtualHost in the Apache configuration. You could do that in Server Templates (you can add arbitrary Apache configuration for each new VirtualHost with Server Templates in the Apache section)…but, you’d also need the Virtualmin AWStats module to know about that, which it doesn’t look like it is configurable in that way. So, some code would need to be written in virtualmin-awstats.
  • Restrict access to Apps
    • NB: you can restrict access to the apps with a .htaccess for example the code below will allow you to block from the internet but allow you local network (192.168.1.0/24) clients to access apps.
      # RESTRICT ACCESS TO DIRECTORY BY IP ADDRESS
# Include in .htaccess of any directory
<RequireAny>
    Require all denied
    #Require ip 1.2.3.4
    #Require ip 5.6.7.8/12
    
    # If local server access to the directory is required
    # add the following; include the server IP addresses (IPv4 & IPv6)
    Require local
    Require ip 192.168.1.0/24
    #Require ip 2001:0db8:85a3:0000:0000:8a2e:0370:7334
</RequireAny>
    • Add into the .htaccess a password requirement
    • Also you could add a referer requirement (I have not tested the code below)
      <IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteCond %{REMOTE_ADDR} !=10.0.0.1
    RewriteCond %{REMOTE_ADDR} !=10.0.0.2
    RewriteCond %{HTTP_REFERER} !=wordpress\.com [NC]
    RewriteCond %{HTTP_REFERER} !=google\.com [NC]
    RewriteRule ^(.*)$ - [R=403,L]
</IfModule>

Serverwide Apps

When you have multiple clients and they all want to be able to use phpMyAdmin, you or they (if allowed) can individually install phpMyAdmin onto their virtual server, however this seems a waste of resources and can lead to Apps being out of date so in this section my aim is to create a central location of all apps that will be used by clients serverwide. This allows one instance to be used and easily kept upto date by the server admin. Apps that are particular to a client can be installed on to their virtual server as normal.

Selecting One Location for your Apps

Use this table to decide the best location for putting your centralised Apps.

Location Pros Cons Who Should use this
www.example.com/apps/
  • No additional virtual server is required
  • Suitable for a low traffic and low resource environment
  • The apps are on your business website and will share it's PHP version
  • Additional traffic to your business site will make tracking more difficult for SEO and SEM purposes
  • Proxying apps on this setup might have some issues
  • Personal Servers
  • Hosting with a few sites
apps.example.com
  • Suitable for high traffic
  • Apps are separate from your business site
  • Proxying apps on this setup is easy
  • Can change the port number of the server for better security
  • An additional virtual server is required which uses more resources
  • High traffic servers
  • Professional Hosting
other.example.com/apps/
  • Suitable for high traffic
  • Apps are separate from your business site.
  • Proxying apps on this setup is easy
  • Can change the port number of the server for better security
  • An additional virtual server is required which uses more resources
  • Having the apps in a sub-folder is a bit pointless when apps have their own virtual server
  • High traffic servers
  • Professional Hosting
/usr/share/
  • No additional virtual server is required
  • Suitable for all traffic types
  • Apps are separate from your business site
  • Alias rule can be added into the Global Apache Configuration and will apply to all Virtual Servers
  • You cannot proxy this App
  • You must use the Apache 'alias' directive and the other required settings and know how they work
  • I am not sure how or if you can control ports with this
  • All traffic types
  • Professional Hosting

Alternative Access Methods

In this section I will outline a different way of accessing these centralised apps to give a better client experience or you can just use them as they are.

ProxyPass / Reverse Proxy

This is the modern way of doing things where you have a dedicated server running your app and then you use your webserver to pass website users request to it and then return the respones back to the user without the app server being seen by the user.

  • is there a way of using that ProxyPass command to redirect <client-domain>/phpmyadmin to https://example.com/phpmyadmin. Virtualmin Pro has a ProxyPass rule builder in it which might do the job.
  • This is not really one location for apps, but it could be, you dont have to proxy whole domains but can proxy folders only.
Apache Alias
  • This will only work if the assets are on the same physical server. I am not sure if it works between virtual servers but it probably does.
  • Integrate phpMyAdmin into the virtualmin GUI - #6 by shoulders - Blue Skies - Virtualmin Community 
    # phpMyAdmin default Apache configuration

Alias /phpmyadmin /usr/share/phpmyadmin

<Directory /usr/share/phpmyadmin>
    Options SymLinksIfOwnerMatch
    DirectoryIndex index.php

    # limit libapache2-mod-php to files and directories necessary by pma
    <IfModule mod_php7.c>
        php_admin_value upload_tmp_dir /var/lib/phpmyadmin/tmp
        php_admin_value open_basedir /usr/share/phpmyadmin/:/usr/share/doc/phpmyadmin/:/etc/phpmyadmin/:/var/lib/phpmyadmin/:/usr/share/php/:/usr/share/javascript/
    </IfModule>

</Directory>

# Disallow web access to directories that don't need it
<Directory /usr/share/phpmyadmin/templates>
    Require all denied
</Directory>
<Directory /usr/share/phpmyadmin/libraries>
    Require all denied
</Directory>
  • One central copy of phpMyAdmin - #6 by mrwilder - General Discussion - Virtualmin Community
    1. Install phpMyAdmin at /usr/share/phpMyAdmin
    2. Webmin --> Servers --> Apache Webserver --> Default Server --> Aliases and Redirects --> Document directory aliases: Add a matched pair with
      • From: /php
      • To: /usr/share/phpMyAdmin
    3. Apply changes and restart Apache.
    4. Now you can go to any domain on the box and get phpMyAdmin, eg:
    5. Make sure you have the authentication mode set the way you intended in (sounds like you want http probably) in config.inc.php!
Redirect (htaccess)

This is a fairly easy way, when a user goes to a particular URL Apache redirects the request and browser to the new URL.

Server Template (Apache Directive)
  • Virtualmin --> System Settings --> Server Templates --> Edit template --> Edit template section: Website for domain --> URL for webmail redirect
    • This will redirect webmail.yourdomain.com to the new URL you define with this setting. 
      /etc/apache2/sites-available/example.com.conf

ProxyPass /.well-known !
RewriteEngine on
RewriteCond %{HTTP_HOST} =webmail.example.com
RewriteRule ^(?!/.well-known)(.*) https://example.com:20000/ [R]
RewriteCond %{HTTP_HOST} =admin.example.com
RewriteRule ^(?!/.well-known)(.*) https://example.com:10000/ [R]

Adding a Virtualmin Dashboard Menu Item

Now we have setup our apps and decided how we are going to access them we should now add a menu link into the Virtualmin Dashboard. Check out the Custom Menu Links section on how to add a menu item

Webmin Modules (3rd Party)

This is just a collection of Webmin modules that do not fit under any of the other categories.

CLI and Commands

The Command Line is very powerful and can do things the GUI cannot, in particular it is ideal for mass changes and automation.

  • Command Line API – Virtualmin
    • Using the command-line scripts included with Virtualmin to manage users, aliases, servers, databases and resellers.
      • Virtualmin includes a script named virtualmin that can be run from the Unix shell to perform actions that are usually done from the web interface.
  • To get a full ist of commands, run these from the Terminal.
    virtualmin
webmin --list-commands
  • config-system – Virtualmin - The config-system command configures a system and its services for use by Virtualmin. It is invoked during Virtualmin installation, but may be invoked later to either configure a new service (assuming the relevant packages have been installed) that was not enabled during installation, or to correct installer issues after they've been fixed by a new version of the virtualmin-config package.
  • bash: virtualmin: command not found
  • modify-php-ini (Changes PHP variables for some or all domains)
    • modify-php-ini | Virtualmin — Open Source Web Hosting Control Panel - Changes PHP variables for some or all domains (virtual servers)
    • This command will add/change the PHP values for CGI, FastCGI and PHP-FPM on a per virtual server basis.
    • This will not change the global PHP configuration files (php.ini / .conf)
    • Examples 
      virtualmin modify-php-ini --domain example.com --ini-name memory_limit --ini-value 128M
virtualmin modify-php-ini --all-domains --ini-name memory_limit --ini-value 128M
    • The files that are altered are (not all listed): 
      # PHP-FPM
/etc/php/7.4/fpm/pool.d/1231231231231234.conf

# CGI, FastCGI, mod_php
/home/username/etc/php7.4/php.ini

# Apache Configuration
if mod_php is present, there can be some Apache configuration changes made. This is probaby to .htaccess files not Apache Directives.
      • if mod_php is present, there can be some Apache configuration changes made.
    • Stuck in termainal after running the help command: virtualmin help modify-php-ini
      • use q or :q to exit from the terminal
    • Virtualmin modify-php-ini - Some clarification needed - #11 by shoulders - Virtualmin - Virtualmin Community
      • `virtualmin help modify-php-ini` what Apache configuration files does this change if any?
  • Rebuild httpd.conf with all virtual hosts - Help! (Home for newbies) - Virtualmin Community
    • You’ll need to get a default httpd.conf in place first, as Virtualmin will definitely be unhappy without one existing.
    • After that, I suspect you could disable the domain and then re-enable, though it might choke on the missing VirtualHost section. Try with one, first, and if it works as expected you can use the List Virtual Servers --> Update or the CLI command to do the rest in bulk.
    • Next, the following code should be put to a file and executed as a script (or run from the console using \) 
      #!/usr/bin/env bash
doms=`virtualmin list-domains --name-only --no-alias`
for dom in $doms; do
   virtualmin disable-feature --domain $dom --web --ssl --logrotate --webalizer --virtualmin-awstats --virtualmin-dav
   virtualmin enable-feature  --domain $dom --web --ssl --logrotate --webalizer --virtualmin-awstats --virtualmin-dav
done
    • You can use the --all-domains switch now I think.

User Restrictions

  • Disable root login
    • TL;DR = no, but remove it from SSH.
      • Leave root enabled but without SSH permissions, making sure you have a secondary account with full sudo permissions.
      • This allows you to perform all tasks in Virtualmin and on your server without using the root account, but should this user account become locked you can still go to the console and access the system.
    • Should I disabled the root account after I have installed Virtualmin - Help! (Home for newbies) - Virtualmin Community
      • You don’t need root for Virtualmin. You need either a root user or an account that has sudo ALL privileges.
      • But, I would make sure you can login with that other user and that other user can do everything in Virtualmin before disabling root login.
      • Disable root from SSH, but leave it on so you can access from the console (and through VM) if you get locked out.
      • You do have to have a root user (many processes start with UID 0), but you can disable direct logins as root in a variety of ways. Using the “lock” option in passwd, as you mentioned above, is one (this sets the hashed password to start with !, which will never match a hash and thus prevent all authentication as this user). Disabling root logins in ssh is another (console root login still works). I tend to prefer the latter, as I like knowing I can get in on the console in the event everything else fails. But a sudo-capable user works for that, too, and you probably always still have single user mode, if you can get to the console.
    • webmin - How to disable root login on Virtualmin - Stack Overflow
      • Different options to remove the root user from Webmin, goto Webmin --> Webmin Users
        1. delete the root account (not recommended)
        2. remove all privileges from the root user.
        3. Limit access by root. Expand Security and Limit Options and select Only allow from listed addresses for IP Access Control. Enter the loopback address(127.0.0.1) into the field. The root user will still exist, but will be unable to login.
        4. Click root and rename it to a new username you will use.
    • To disable root login in Virtualmin, you can do one of the following | Bing Search
      1. Create a new sudo user via command line. That user can then login to webmin with all privileges. Once confirmed that you can login to webmin with new name, remove root password via command line or disable root login via webmin
      2. Log in as a user with administrator privileges and click on Webmin --> Webmin Users. In the list of users, either delete or remove all privileges from the root user.
      3. Give an existing user sudo privileges, and they can then log into Virtualmin as the Master Admin. You could also just change the shell on the root account, so they can't log in via SSH and such, but can log into Virtualmin.
      4. Locate the Allow login by root option and select the radio button next to No.
      5. Edit the /etc/ssh/sshd_config file and uncomment (if it is commented) the directive PermitRootLogin and set its value to no.
      6. You can also set PermitRootLogin to “without-password”, which says that you can login remotely as root, but only if you’re using an SSH key.
  • File Manager - Users are able to access all files on server, change this setting
    • Webmin --> Usermin Configuration --> Access Control Options --> Root directory for file chooser: "User's home directory"
  • SSH
    • SSH Access to Website - #10 by danwtsa - Virtualmin - Virtualmin Community
      • Additional users in the domain share a group with the domain, but not the same user. So, the public_html directory would need to be writable by the group for a user with a different UID (but the same GID) to write to it. There are some security implications to making that change, but if all users in the domain are trusted to have write access to the website, you should be fine.
      • You can alternatively create an FTP user (and allow them to also use ssh), in the Edit Users page, which I believe will share a UID with the domain owner user. I need to look at how things have changed, as I don’t actually know what all the user types do now, after Ilia renamed them all. (I’m sure it’s less confusing than it was before, at least I hope so, but I knew what the old ones meant and don’t know what the new ones mean.)
  • Misc
    • Cannot change 'allowed login type' in owner limits?
    • The virtual server owner will always have access.

Templates / Defaults

General

  • Server Settings and Templates – Virtualmin
    • Variable subsitutions which can be used in the text boxes below, which will be evaluated at server creation time.
    • An overview fo the different templates.
  • Template Variable Listing – Virtualmin
    • This page lists many of the template variables that you can use in email messages to domain owners and mailboxes, initial website content (Skeleton), Apache and BIND configurations and many other places in Virtualmin and these variables will be parsed out and the relevant values substituted, very powerful.
    • When the template is related to a sub-server, variables for the parent server are also available with PARENT_DOMAIN_ prefix, like ${PARENT_DOMAIN_HOME} and ${PARENT_DOMAIN_DOM}

Skeleton directory for files / Default website files / Holding page

  • Located here: /etc/skel
  • The Index file must be located as so: /etc/skel/public_html/index.html
  • You can have different default files setup in different skeleton folders allowing for mulitple options. You just need to change where a particular Server Template gets the skeletong files from.
  • A skeleton directory contains files that will be copied into the newly created home directory of the domain user. It can contain other directories, which will also be created in the home directory. This can be used to provide a pre-configured set of scripts or web content for some or all server templates.
  • Variable substitution in skeleton HTML files
    • When you make your templates, you can include in them variables taht will be replaced by user information and so on.
    • Just for reference, the most commonly used variables are documented here: Server Settings and Templates – Virtualmin.
  • Website Default Page – Virtualmin
    • Upon setting up a virtual server using Virtualmin, a default landing page is created. This page serves multiple purposes and provides information about the status of the website and server configuration in general.
    • These might not appear if you have files in your skeleton directory.
  • Is it possible to change the default page shown when virtual site are disabled | Virtualmin
    • You can configure that at: Virtualmin --> System Settings --> Server Templates --> e.g. Default Settings --> Website for domain
      • Disabled website HTML: This field can be used to customize the message that appears when connecting to a website for a disabled Virtual server. The default message simply states Website Disabled.
      • Disabled website URL: This option can be used to re-direct browsers connecting to the website of a disabled virtual server to a completely different URL, rather than simply displaying a locally served HTML message.
  • How can i change the default "LandingPage" (webserver home page using HTML on webmin/virtualmin - Webmin - Virtualmin Community
    • Add a new index.html to /etc/skel/public_html/ (or whatever skel directory you have configured).
    • You should not change anything in the Virtualmin installation directories. Any such changes will just be overwritten next time you install an update.
    • I guess we should clarify whether you want to edit one such file (one already setup for a domain account in the user’s public_html directory), which is what Stegan is talking about, or if you want to change the default that is put in place in new domains, which is what I’m talking about with adding a file to /etc/skel/public_html.
    • Skeleton dir is also configurable per-Server Template (under Home Directory), so you can have multiple default pages, if you want.
    • You can put anything you want in your own default page. There are literally no restrictions imposed by Virtualmin on what you put in /etc/skel/public_html/index.html. If you want it to be a redirect, make it a redirect (that seems confusing, to me, without a little explanation, though…I think your users would be better served by a little bit of explanation of how to replace the default page with their own content, etc.).
  • Server Template - Skeleton Substitution - Help with Speed - Virtualmin - Virtualmin Community
    • Take a peek in Virtualmin --> System Settings --> Server Templates --> Default --> Home directory
    • Jamie recently added a feature that that would allow you to specify a regex for file patterns to not perform the substitutions.

Server Templates

  • Create a Server Template (notes)
    • The 'Default Settings' template cannot be deleted.
    • These are used for the initial build of a Virtual Server and various Post-processes such as creating a database and resetting DNS Zones.
    • Changes are not actively reflected to accounts using the template.
    • cPanel does not have an equivalent to this. This is server level administration and allows setting up pre-determined server configurations.
    • For use by: in templates you create you get 4 options where as the default templates only have one option here.
    • You cannot clone the 'Default Settings' template, but there is a button called 'Create a template from the default settings'
    • The 'Create a template from the default settings' refers to the 'Default Settings' template. I am not sure why it does not have a clone button which would be more logical.
    • You can edit and save the 'Default Settings' and 'Settings for Sub-Servers' templates.
    • When you select use 'From default settings ', these settings are inherited from the 'Default Settings' server template.
    • If you choose 'Create a template from the default settings' a new template is created used all of the settings from the default template rather than setting everything to default these are hard settings.
    • If you choose 'Create an empty template' a new template is created with all options set to default.
    • Not all of the settings are used upon the creation of a Virtual Server such as the 'MariaDB database' settings, these are only used when you create a new database. This means that the settings in the various different sections are used at different times.
    • I cannot tell if any of the settings are used in a live fashion or they are only used when new items are created, which is definitely the case for most if not all of them.
  • Create a Sub-Server Template (notes)
    • The 'Settings for Sub-Servers' template
      • Cannot be deleted.
      • Inherits the default settings from the 'Default Settings' template, not the parent server's template.
      • Will inherit default settings from it's yet to be determined parent template (from the parent virtual server / top-level virtual server / primary server that the user is part of). There is no-multi-dimensional inheritance occurring.
      • Can only be used as a template for a sub-server.
      • Can be cloned.
      • If you clone this template, all the 'For use by' options are available.
      • Sub-Server templates only really work if they do not have mail, and the DNS is managed by the parent so the inheritance from the 'Default Settings' rather that the parent's template does not become and issue. When it does you must make copies of the 'Settings For Sub-Servers' template and work on them instead of a single template for Sub-Servers.
    • These are just like a normal Server Template Except:
      • When you select use 'From default settings ', the settings are inherited from parent template (from the parent virtual server / top-level virtual server / primary server that the user is part of). There is no-multi-dimensional inheritance occuring.
      • Not all categories are available (or should not be) i.e. 'Administration user'.
      • Can only be used as a template for a sub-server, unless the other types are ticked in 'For use by' and if used in one of the top level template roles, the inheritance will change it's source from the parent template to the 'Default Settings' template.
  • Import / Export Server Templates
    • Import / Export of server templates between 2 physical servers - is it safe - Virtualmin - Virtualmin Community
      1. is it safe to manually copy templates from /etc/webmin/virtual-server/templates and then copy them into the same folder on the remote system?
        • Yes, it should work just fine, as long as a template is stored (saved) as a file.
      2. If I use the backup and restore mechanism as outlined here Separate function to Backup Server Templates [#19325] | Virtualmin 2 will this overwrite all of the current templates in the target system when I restore?
        • Yes, it should be overwritten upon restore; that is the purpose of the restore function.
      3. Where is the Default Template stored as it does not seem to be in this folder.
        • If it isn’t saved, then Virtualmin relies on the defaults primarily defined in the /etc/webmin/virtual-server/config file.
    • Separate function to Backup Server Templates [#19325] | Virtualmin
      • You can actually backup just templates by:
        • Virtualmin --> Backup and Restore --> Backup Virtual Servers
        • In the "Servers to save" section, choose "Only selected" but don't select any domains from the list.
        • Then in the "Features and settings" section in the "Virtualmin settings to also backup" field, check all the boxes for global settings, check "Server templates and plans".

Account Plans

  • How to Setup a New Account Plan in Virtualmin | Hostwinds - What is an Account Plan in Virtualmin? Like packages in cPanel, Account Plans allow you to customize specific settings a user has access to based on their assigned plan.
  • Create an Account Plan (notes)
    • These control things like: Permissions, Features, Bandwidth and Disk Quotas.
    • Is equivalent to cPanel Packages + features if they were combined.
    • There are some issues with layout and settings matching between 'Edit Owner Limits' and 'Account Plans'
    • The 'Save and Apply' button will save the settings and then push them to all members of the plan.
      • This allows you to dynamically update plan memeber's settings without visiting each one individually.
      • This will override permissions set in Virtualmin --> Administrative Options --> Edit Owner Limits
        • 'Edit Owner Limits
          • This is where account stores these settings
          • The name is a bit misleading.
    • The 'Save' button will just save the template, no changes will be pushed to members.
    • A Sub-Server cannot have an account plan assigned to it. Account Plans can only be associated with the top-level Virtual Server.
    • A Sub-Server shouldn’t have an account plan associated with it, Account Plans should only be associated with the top-level Virtual Server.
  • Settings
    • Basic plan details
      • All settings can be pushed.
      • Pushed to: Virtual Server --> Manage Virtual Server --> Edit Owner Limits --> Virtualmin limits for server owner
      • Quotas
        • Account plans confusion - Virtualmin - Virtualmin Community
        • Quota refers to disk space.
        • Quota for entire server
          • includes other users created by the domain owner, e.g. mail users homes, too. (This is implemented by setting the group quota for the domain group, which all users in the domain are a member of.)
          • My interpretation scenarios:
            1. Top-Level Server + Sub-Servers
            2. Resellers + Their Clients
        • Quota for server administrator user
          • is the quota for the domain owner account (a user quota), and will apply to website content, database content (if databases are on the same filesystem as /home), etc.
    • Allowed virtual server features
      • Most settings can be pushed.
      • Pushed to: Virtual Server --> Manage Virtual Server --> Edit Owner Limits -->Allowed capabilities and features --> Allowed features for servers
      • Some options can only be selected here and used on the creation of a virtual server
        • Administration user
        • Home directory
      • What is ‘Allowed virtual server features’
        • These settings define what services are enabled for the Virtual Servers when the ‘Account Plan’ is applied, but they can be overridden when a user utilises ‘Edit Owner Limits’, perhaps this is why the notes keep referring to default settings.
        • These are permissions. They do not enable or disable services.
      • Default available features
        • Tooltip: When this option is set to Automatic (as it is by default), new top-level virtual servers will have their allowed features set based on those initially enabled when the server is created.
        • Tooltip Translation: if the service is enabled on Virtualmin, then enable the related permissons.
    • Allowed Capabilities
      • All settings can be pushed
      • Pushed to: Virtual Server --> Manage Virtual Server --> Edit Owner Limits -->Allowed capabilities and features --> Edit capabilities for virtual servers
      • Default editing capabilities
        • Automatic
          • Tooltip: If the Automatic option is selected, limits are determined based on whether the virtual server owner is allowed to create sub-servers or not (controlled by the Limit on number of virtual servers field). If so, he will have access to all capabilities. Otherwise, he can only manage users, aliases and edit web pages.
          • Tooltip Translation:
            • If the Virtual Server Owner can create sub-servers, enable everything, and if not, only enable:
              • Can manage aliases
              • Can Manage users
              • Can change domains password
            • He can also edit web pages.
          • Basic Questions on VirtualMin,WebMin,UserMin - #6 by JosephV - Help! (Home for newbies) - Virtualmin Community
            • This is an old post but might still be true, in the sense that above happens and then these tests are applied to then reduce functionality.
            • What “Automatic” means in that case is that it tries to determine what you’d want, based on other limits that are setup throughout the Server Templates and Account Plans.
            • For example, there’s a capability called “Can manage aliases?”.
            • Just above that in the Account Plan details, is an option named “Limit on number of aliases”. If the limit was set to 0 (meaning there are no aliases allowed), the “Automatic” setting would assume that the Virtual Server owner isn’t allows to manage aliases, and won’t display the option.
            • That’s just one simple example – there’s similar options scattered around the Server Templates and Account Plans.
        • Explain some options in Account Plans - Virtualmin - Virtualmin Community
          • Administration user
            • They’ll have a Virtualmin login that can manage the website(s) associated with the account.
          • Home directory
            • Exactly what it sounds like. They’ll have a home directory, which is necessary for serving any kind of web content or application or accepting mail or pretty much anything else.
          • Realistically, you pretty much always want both to be enabled, for any normal use of Virtualmin. Aliases don’t need/get either, and I’m guessing that’s kinda where the variability comes from.

Backup, Restore and Migrations

  • Backup and Restore
    • How to setup automatic MySQL database backups with Virtualmin - Virtualmin has an essential feature that can be used to enable automatic database backups. This tutorial shows how to enable weekly backups with Virtualmin.
    • Backup Virtual Servers: Download Via Link - #3 by cyberndt - Virtualmin - Virtualmin Community
      • Q: The option “Download Via Link” creates the backup and gives you a link for that download. I am asking where is the backup stored on the server? Does it get created in an directory? or, is it only a /tmp file?
      • A: it’s in the directory /tmp/.webmin/ until the link is clicked then it’s erased
    • Backup and Restoration – Virtualmin - Virtualmin provides multiple tools to help you keep good backups automatically. The first step after any installation of Virtualmin should probably be thinking about your backup procedures and setting up Virtualmin to automate those procedures for you.
    • Backup and restore (CLI) – Virtualmin - Virtualmin has the ability to backup and restore virtual servers either manually or on a set schedule, using the web interface. However, you can also use the command line programs listed below to make backups. This can be used for doing your own migration to other systems or products, or manually setting up custom backup schedules for different servers.
    • Backup and Restore for Webmin-Virtualmin VPS | Full Circuit | Elegant Solutions to Difficult Problems - How to backup and restore a website VPS using free Webmin/Virtualmin with s3cmd and Amazon S3 storage.
    • Backup Configuration Files | Webmin
      • Webmin --> Backup Configuration Files
      • Most Webmin modules work by editing configuration files on your system. Each module knows which configuration files it manages, and what commands need to be run to activate them. Not all modules actually deal with config files though - for example, the Database Server modules work by executing SQL commands. As such, it cannot participate in the configuration backup process.
      • The Backup Configuration Files module can collect information about config files from other modules, and create and restore backups containing some or all of those files. It is designed for saving the configuration of a single system, but not for migrating configs from one server to another - that would be far more complex.
    • Google Drive backups - #3 by apt_virtualmin - Help! (Home for newbies) - Virtualmin Community
      • rclone example: 
        rclone sync /your-local-backup-dir gdrive:/your-google-drive-path/
      • Virtualmin Pro supports Google Drive natively.
    • Follow symlinks when making backup - Virtualmin - Virtualmin Community
      • Q: Is it possible to set up the Backups module of Virtualmin to follow symlinks? I have part of my web site pointing to a mounted drive via a symlink and currently it doesn’t follow and backup those files.
      • A:
        • Virtualmin uses tar to make backups. By default, tar does not dereference symbolic links, meaning it archives the link itself rather than the file or directory it points to.
        • Luckily, you can change this behavior by passing to tar additional -h or --dereference option with
          • Virtualmin --> System Settings --> Virtualmin Configuration --> Configuration category: Backup and restore --> Additional parameters to tar command
  • Setting Locations
  • Backup File
    • Where are the SQL files?
      • The databases are in the root of the archive and have a file extension starting with COM_MYSQL_ and aformat of COM_MYSQL_MYDBNAME
  • etckeeper
  • Migrations
  • Databases
    • Webmin --> Servers --> MariaDB Database Server --> Backup Databases
      • Click this button to setup the backup of all MariaDB databases, either immediately or on a configured schedule.
      • There is a configuration page when you click this button.
  • Error: No route to host
    • This error is caused when you do not have the DNS correctly set for your server's hostname. 
      Fatal Error!
Restore failed : Failed to transfer file : Failed to connect to dev.........uk:10003 : No route to host
    • You will still get this when you have the following checkboxes selected

Networking

  • NAT
  • Change Hostname
    • Webmin --> Networking --> Network Configuration --> Hostname and DNS Client --> Hostname
    • How To Change The Hostname In Virtualmin | Hostwinds - Typically, to change your server's hostname, you'd need to login to your server via SSH and issue the hostname command followed by the new hostname. However, with Virtualmin, you can actually change the hostname by using the Hostname and DNS Client module. This article focuses on teaching you how to locate this module and change your server's hostname.
  • DNS server = 127.0.0.53 ?
    • Webmin --> Networking --> Network Configuration --> Hostname and DNS Client --> DNS servers
    • see the systemd-resolved (DNS Resolver) section.
  • Nameservers
    • Changing default nameservers - Help! (Home for newbies) - Virtualmin Community
      • You can edit the nameservers used by Virtualmin for new domains in:
        • Virtualmin --> System Settings --> Server Templates --> Default --> DNS Domain
      • To edit NS records for an existing domain from within Virtualmin, you’d need to go into:
        • Virtualmin --> DNS Settings -> DNS Records
      • It’s possible to make changes to all DNS records at once by using the command line tools.
        • To see the available options, you can run “virtualmin modify-dns” fromt he command line.
        • You’d likely need to first run a command to remove the “NS” records, and then run another command in order to add the new ones.
    • DNS Frequently Asked Questions – Virtualmin
      • Virtualmin error: 127.0.0.1 isn't listed in /etc/resolv.conf
      • How do I setup nameservers for my server?

Locations of

To save spending ages re-finding files and other things i have made a litte collection of locations here to help.

Repositories

Files

  • Webmin
    • Code/.pl/.cgi
      • /usr/share/webmin/
      • /usr/share/webmin/webmin/
    • Webserver
      • /usr/share/webmin/miniserv.pl
    • All Webmin configuration files
      • /etc/webmin/
    • Settings
      • /etc/webmin/webmin
  • Virtualmin
    • Code/.pl/.cgi
      • /usr/share/webmin/virtual-server/
    • Webmin module settings
      • /etc/webmin/virtual-server/
    • Server Templates
      • /etc/webmin/virtual-server/templates/<template_id>
    • SSL (when not in user's directories) Per-domain directory under
      • /etc/ssl/virtualmin
    • Main config file
      • /etc/webmin/virtual-server/config
    • Server Template Wizard
      • Wrapper
        • /usr/share/webmin/virtual-server/edit_tmpl.cgi
      • Individual Section Templates (this builds the forms and tables)
        • Website for domain + PHP options: /usr/share/webmin/virtual-server/feature-web.pl
        • Mail for domain: /usr/share/webmin/virtual-server/feature-mail.pl
        • Spam filtering: /usr/share/webmin/virtual-server/feature-spam.pl
      • Example modification of a Server Template option
    • Virtualmin Internal Default Holding page(s) template (eg Domain default page)
      • /usr/share/webmin/virtual-server/default/
    • Virtualmin Pro Subdirectory (Commercial Code)
      • /usr/share/webmin/virtual-server/pro/
  • Usermin
    • Code/.pl/.cgi
      • /usr/share/webmin/usermin/
    • Settings
      • /etc/usermin/
    • Webmin module settings
      • /etc/webmin/usermin/
  • Authentic Theme
    • Code/.pl/.cgi
      • /usr/share/webmin/authentic-theme/    
    • Webmin module settings
      • etc/webmin/authentic-theme/
    • Manifest template
      • /usr/share/webmin/authentic-theme/manifest.template
    • Built manifest file
      • /etc/webmin/authentic-theme/manifest-webmin.json
  • Services
    • BIND Zone files
      • /var/lib/bind

SSL Certificates / Lets Encrypt (LE)

  • General
    • Virtualmin --> Manage Virtual Server --> Setup SSL Certificate
    • If your domain is not pointing to your Virtualmin server, then a Lets Encrypt Certificate will not be requested and it will have to be done manually later.
    • Once you have manually added an Lets Encrypt Certificate, Virtualmin will keep it updated via one fo the CRONs
    • LE Cert = Lets Encrypt Certificate.
    • How to add an SSL certificate – Virtualmin - These instructions will tell you what fields to fill in, if not already.
    • SSL and Virtualmin – Virtualmin
    • Free SSL Certificate (Lets Encrypt) – Virtualmin - This page will provide instructions for requesting a Let's Encrypt SSL certificate in Virtualmin.
    • Challenge Types - Let's Encrypt
      • HTTP-01 challenge: Validation by using your website.
      • DNS-01 challenge: Validation by DNS entries. This is required for creating wildcard certificates.
      • TLS-SNI-01: depreceated
      • TLS-ALPN-01: The challenge is done over TLS.
    • For anyone having issues with certificates expiring, you can run the following command on the server to get a list of certificates sorted by expiry date. 
      virtualmin list-certs-expiry --all-domains
  • Settings for enabling Lets Encrypt Certificates
    • Virtualmin --> System Settings --> Virtualmin Configuration --> Configuration category: SSL Settings --> Show Let's Encrypt error at domain creation time?
      • Tooltip: When set to Yes, Virtualmin will attempt to request a valid Let's Encrypt SSL certificate for new virtual servers. This will only succeed if they have a domain name which is resolvable from outside your system, so that it can be looked up by the Let's Encrypt service.
      • This options needs to be on
    • Virtualmin --> System Settings --> Virtualmin Configuration --> Configuration category: SSL Settings --> Request Let's Encrypt certificate at domain creation time
      • Virtualmin will do (by default) a connectivity check before even requesting a SSL certificate from Lets Encrypt. This extra check can be disabled with 'Yes and skip connectivity check'.
    • Virtualmin --> System Settings --> Virtualmin Configuration --> Configuration category: SSL Settings --> Setup Let's Encrypt SSL certificate for hostname
      • This allow you to add an SSL for the servers hostname (ie. the domain name you gave to your Virtualmin server eg: server.example.com).
      • This feature in Virtualmin sets up a default domain with your hostname. This domain is hidden and doesn't serve any special function. It's there to improve your experience by ensuring you can log into Virtualmin with a valid SSL certificate right after installation.
      • Virtualmin --> System Settings --> Re-Check Configuration
        • This is required to apply any changes of the `Setup Let's Encrypt SSL certificate for hostname`option.
        • Options
          • Yes, and keep visible = A virtual server will appear in your list of virtual servers and stay there allowing you to edit it as a normwal virtual server.
          • Yes = This just presents the domain while it is doing the Let's Encrypt SSL handshakes and then hides it again.
          • No = no LE SSL Certificate.
      • You can actually use any virtual server and the correct port to access Virtualmin, using that Virtual Server's SSL certificate so you do not actually need a real SSL on your hostname.
      • How to get LE certificate for the now hidden host? - Virtualmin - Virtualmin Community
      • Let's Encrypt certificate for Virtualmin host itself? [SOLVED] - #2 by calport - Webmin - Virtualmin Community
        • Why? Just login to Webmin on the hostname of one of your Virtualmin managed domains. Webmin will use the cert for that domain name.
        • Webmin can request Let’s Encrypt certs for itself in Webmin the Webmin SSL configuration page, but it’s trickier, since it has less certainty about how things are setup than Virtualmin does.
        • create a virtual server with the hostname of the Virtualmin server
          • I think this is automatically done during Virtualmin installation, if it can be done and the hostname resolves. (This automatic domain is a “free” domain for Pro users. And it can’t have mail, for some technical reasons that are the same reasons we tell people don’t name your server the same as a domain you’ll be managing in Virtualmin.)
          • I’m ambivalent about whether this is a good feature (mostly leaning toward “not a good feature”, but Ilia and Jamie like it, so it stays). I think I prefer keeping things simple and just using Virtualmin domain names to login to Virtualmin. Then you don’t have to ever think about the name of the server itself…which is mostly irrelevant.
  • If a Lets Encrypt SSL Certificate is not created when you create a Virtual Server, but instead you get a self-signed one instead. All of the settings are correct and you have got no warnings, what is the issue?
    • If your domain does not resolve to your server, you will not get a Lets Encrypt certificate because validation will fail.
    • If you do have Show Let's Encrypt error at domain creation time? you will not get any error messages about this.
    • If the LE cert fails at domain creation, then you have to manually enable it in the Virtual server after the fact, and then it will stay automatic
  • Enable Wildcard for a domain
    • Virtualmin --> Web Configuration --> Website Options --> Website matches all sub-domains
      • Tooltip: If the virtual server's DNS domain is hosted on this system, Virtualmin will also add the wildcard * DNS record when Yes is selected.
    • How to add a wildcard or multi-domain SSL certificate – Virtualmin
    • Let's Encrypt wildcard certificate - Virtualmin - Virtualmin Community
      • You cannot validate for a wildcard certificate without using DNS validation. And, you can’t use DNS validation if you aren’t managing DNS with Virtualmin.
      • You generally should not use wildcards. They have security implications on top of being more difficult to validate, if you’re not hosting your own DNS.
      • A website cannot be used to validate a wildcard cert with Let’s Encrypt.
    • Configure Wildcard Certificate using LetsEncrypt and ACME - #26 by reigningking - Help! (Home for newbies) - Virtualmin Community
      • Joe
        • First up: You almost certainly should not use wildcard certs. They have security implications and are more difficult to validate. There is no reason to use them, you can get as many certs as you need for all the domains and subdomains you’ll be using, no reason to use wildcards in the vast majority of cases.
        • Second: You decided in your DNS propagation thread to not host DNS on the Virtualmin server. So, Virtualmin cannot request wildcard certificates for you, because it requires DNS validation to get a wildcard cert from Let’s Encrypt (that’s the only way to prove you own the zone and not just one name in the zone).
        • You can have Virtualmin create a certificate for every subdomain, assuming Virtualmin is managing every subdomain. If it is merely an alias and you’re application decides what to serve based on the name, that can still work in Virtualmin without DNS validation…just add all the aliases to the certificate for the domain. A regular (non-wildcard) certificate can have a bunch of names associated with it, and Virtualmin will offer to do that for Aliases, and it should work fine assuming you have DNS working correctly for all those names.
        • If you must use a wildcard, just use the certbot standalone mode, in interactive mode. I don’t know what that script could do to make that workflow easier.
      • I successfully did this using ACME SH. I followed this help here: dnsapi · acmesh-official/acme.sh Wiki · GitHub
    • Let's Encrypt DNS challenge for wildcards - Virtualmin - Virtualmin Community 
      • certbot -d domain.tld --manual --preferred-challenges dns certonly
      • Joe: Oh, as always (and for the same future users who might stumble on this thread), I recommend not using wildcards. They have a variety of security implications, and they’re (usually) harder to validate.
    • Creating Wild Card SSL in Virtualmin - #3 by swelljoe - Help - Let's Encrypt Community Support
      • Joe:
        • Wildcards can only work in Virtualmin if Virtualmin is managing DNS (whether that's locally or in a cloud service like Route 53 or via Cloudmin Services), since it has to update the TXT record to match what LE expects.
        • If you aren't managing your DNS with VIrtualmin, you'll either need to not use wildcards (which I think is generally a good practice anyway...wildcards have some security implications, and are just more annoying to deal with) or manage them using certbot directly, assuming certbot can work with your DNS provider (it has plugins for stuff like Route 53 and several other APIs).
    • Let's encrypt DNS challenge - Virtualmin - Virtualmin Community
      • On Virtualmin’s DNS records I only see one TXT record: 
        _acme-challenge.domain.com TXT
  • http needs to be available for your first LE certificate (maybe not anymore)
    • The reason is that if you do not have a valid SSL certificate and you have enforced https by using HSTS or rewrite then Lets Encrypt will fail the process.
    • If your SSL certificate is valid/truested, i.e.e you are renewing, then HSTS or redirects (http --> https) will cause not issue as LE allows this.
    • HSTS and Let's Encrypt - #4 by schnappijedi - Server - Let's Encrypt Community Support
      • If you have that redirection in place, Let’s Encrypt will respect it and follow it. This means that you don’t need to disable the redirection to perform certificate renewals with Let’s Encrypt. A setup with HTTP → HTTPS redirection, with or without HSTS, is perfectly fine for Let’s Encrypt.
      • or the HTTP-01 validation method, Let’s Encrypt will
        • require an initial valid HTTP response on port 80
        • follow any HTTP 301 redirections, to the same or a different host, in either HTTP or HTTPS protocols
        • ignore any mismatched or expired certificates on HTTPS URIs reached as a result of such redirections
        • ignore the presence of HSTS (that is, the validation always starts with HTTP on port 80)
  • Current SSL Certificate - Buttons
    • Virtual Server --> Manage Virtual Server --> Setup SSL Certificate --> Current Certificate
    • On this page there are some buttons but they just need some clarification
    • Certificate not installed
      • Copy SSL Certificate to Services
        • Install this certificate on this Virtual Server for use by the attached services on this domain, such as email and websites.
        • If Let’s Encrypt is enabled, Virtualmin will automatically install the certificate for you.
        • The description text implies the certificate will only be used for Dovecot, however after reading the options when the certificate is installed, I think this text needs updating.
      • Set as Default Services Certificate - Install this certificate as the Virtualmin Server Default SSL certificate.
    • Certificate installed
      • Remove SSL Certificate from Services - As the description says, it will remove the certificate from all services it has been installed into.
      • Set as Default Services Certificate - As above.
    • Links

Virtual Servers

General

  • Can a Sub-Server Be Created at Top Level? - Virtualmin - Virtualmin Community
    • You can convert a sub-server into a top-level domain.
    • I’m not sure I understand what you’re describing, but I’ll mention the following two things that may be useful to keep in mind when migrating from cPanel:
      • Subdomains are just names in Virtualmin. It doesn’t care. A name is a name. sub.domain.tld can be a top-level domain, or it can be a sub-server of domain.tld or it can be a sub-server of some other domain.tld. It doesn’t matter, it’s a name.
      • Sub-servers are about ownership in Virtualmin. That’s it. A sub-server is owned by some other top-level domain account and lives in a subdirectory within that user’s home (this is a compromise, but it’s to ease administration, permissions, and backups). A subdomain has no technical reason to be a sub-server, and there is no limit on what you can name a sub-server (unless you impose one with configuration).
  • How to Change Virtual Server Owner’s Password | Virtualmin — Open Source Web Hosting Control Panel
    • Virtualmin --> Edit Virtual Server --> Configurable settings --> Administration password

Creating

Moving and Renaming (on server)

  • General
    • When you move a Virtual Server, the files are moved aswell.
  • Sub-domain account type
    • Sub-domains accounts are not sub-servers. They are only created when you import a cPanel archive (by design) and are not the preferred method.
    • Virtualmin for cPanel Users – Virtualmin
      • cPanel is an old, but still very popular, webserver administration tool. Since many new Virtualmin users have only experienced system administration through cPanel, they may find some terms and concepts in Virtualmin new or confusing. This short guide will attempt to point out a few of the gotchas that we've found most commonly trip up former cPanel users trying out Virtualmin for the first time.
      • cPanel has a type of domain account called a "sub-domain", which creates a new virtual host that only provides web service and puts the content into a subdirectory of the document root of the parent domain.
    • Sub-server like a Top-level server - #2 by tabletguy - Help! (Home for newbies) - Virtualmin Community
      • sub-domain account types are deprecated, and were never a good idea…we added them to make a few cPanel users more comfortable, but it confused everybody else
  • Default Sub-domains/Alias
    • When you create a virtual server the following 'sub-domains' are created:
    • These do not count towards your domain limits.
  • Sub-Servers
    • These allow you to add sub-domains or other domains under one Webmin account while maintaining a completely different hosting environment for each of them.
    • cPanel sub-domains all share the same hosting environment.
    • How to create a sub-server – Virtualmin
      • This tutorial will cover how to create a sub-server, allowing for a second domain to be setup within a given Virtual Server account.
      • A sub-server is also the recommended way to create a sub-domain website that is owned by the parent domain. Sub-servers are not limited to sub-domain names, but they work well for hosting sub-domains.
  • Change Domain Owner / Rename Domain
    • You can promote between parent and sub-server
    • You can move a sub-domain between owners
    • Sub-servers share their DNS with their parent. This reduces duplication of DNS records by having a single DNS Zone.
    • Changing the owner's username, this can be done at
      • Virtualmin --> Manage Virtual Server --> Change Domain Name.
    • Transferring a sub-server to another parent top-level server, this can be done at
      • Virtualmin --> Manage Virtual Server --> Move Virtual Server.
      • This page allows you to convert this top-level server into a sub-server under an existing domain.
    • Convert a sub-server to parent
      • Virtualmin --> Manage Virtual Server --> Move Virtual Server
      • Select Convert to parent, and it will…convert the sub-server to a parent (non-sub-server) virtual server
      • This option might only appear when you have at least one sub-server.

Restrictions

  • General
    • Limit what a Server owner can access and configure - Help! (Home for newbies) - Virtualmin Community
      • Account Plans
        • Virtualmin --> System Settings --> Account Plans
        • There’s a number of screens in there that allow you to tweak what exactly a user has access to when you create a Virtual Server for them.
        • You could also make different Account Plans – one with certain options disabled, and another with all those options enabled.
      • SSH
        • As far as SSH goes – the key there would be to make sure users who should not have SSH access don’t have a login shell.
        • To disable SSH by default, you can go into System Customization -> Custom Shells, and look for the shell where both “Admin” and “Default” is set. Chances are, that shell is “/bin/bash” or perhaps “/bin/sh”.
        • Uncheck “Default”, look for the “/bin/false” shell, and make sure it has “Admin” and “Default” checked. This will prevent SSH logins by default.
        • *** You just forgot to mention that I had to check “Enable” in the new custom shell but it was clear anyway
      • Modules
        • Q: everything below the webmin modules is still active. can i disable these for specific server owners?
        • A: Those are configurable within the Server Template
          • Virtualmin --> System Settings --> Server Templates --> Default -> Administrators Webmin modules.
  • Limit Bandwidth / Bandwidth Monitoring
    • Bandwidth Monitoring | Webmin - About The Bandwidth Monitoring module can be used to create simple reports on bandwidth usage by port, host, protocol and time for traffic sent from or routed through your system. It is useful for both stand-alone hosts, and those that act as a gateway (possibly with NAT) for a network.Before it can be used, the module must setup several firewall rules and a syslog entry to capture traffic sent and received via your system.
    • Bandwidth monitoring and limits are extremely resource intensive, by necessity. It has to deal with every packet in and out of the system, so it requires some extra CPU and disk space to work.
    • The Pro version has a feature to email users/clients when certain limits are reached.
    • Enable Bandwidth monitoring
      • Virtualmin --> System Settings --> Bandwith Monitoring
        • Bandwidth monitoring active: Yes
        • Disable servers that exceed limit: Yes
        • Re-enable servers that fall below limit: Yes
        • NB: This page can be used to enable bandwidth accounting for virtual servers, to notify server owners and the master administrator when a server exceeds its allowed bandwidth.
      • To change the bandwidth quota
        1. Log into the control panel (as root)
        2. Choose the Virtual Server in question from the select list
        3. Virtualmin --> Edit Virtual Server --> Quotas and limits --> Bandwidth limit
          • NB: this will only appear if you have Bandwidth monitoring enabled.
    • Bandwidth Usage
      • This will show per domain, per ay the bandwidth usgae
        • Virtualmin --> Logs and Reports --> Bandwidth Graph
  • Disk Quotas
    • Disk quotas are enforced in the GPL version as well as Pro.
    • To change the disk quota
      • Log into the control panel (as root)
      • Choose the Virtual Server in question from the select list
      • Virtualmin --> Edit Virtual Server --> Quotas and limits --> (Total server quota | Server administrator's quota)
  • Edit Resource Limits (Pro only)
    • Virtualmin Professional - Resource Limits | Virtualmin
    • Limited Ressources for customers - Help! (Home for newbies) - Virtualmin Community
      • Q: Is there a solution how i can set the maximum cpu usage or memory for users?
      • A:
        • Virtualmin --> Manage Virtual Server --> Edit Resource Limits
        • You can tweak options for the maximum number of processes, process size, and CPU time.
        • What I believe it does is tweak /etc/security/limits.conf, which is something you can do manually as well. That’s an OS thing, not a Virtualmin thing.
    • How to set CPU & Memory limits for Virtual Servers (PRO)? - Help! (Home for newbies) - Virtualmin Community
      • just purchased the Pro version of Virtualmin and based on Googling was hoping to be able to limit user’s CPU and Memory usage. I can’t seem to find any of the options, so my question is… where can I find the options to limit how much CPU & Memory each Virtualmin user can consume?
      • Virtualmin --> Manage Virtual Server --> Edit Resource Limits --> Resource Limits
      • Q: Is there anything else I should / could check in order to activate the feature?
      • A:
        • In order to have this feature displayed:
          1. You need to be a master admin
          2. Your OS type should be set correctly as linux on Webmin config (cat /etc/webmin/config | grep os_type)
          3. There should be a file on your system called /etc/security/limits.conf, meaning the package libpam-modules must be installed
          4. You must be able to edit a domain and a domain must have a correspondent unix user (can be checked in a domains configs under /etc/webmin/virtual-server/domains by finding the domain config file and checking for unix=1 option).
        • All of this is the case of default installation. If you’re missing something try to remember what you have changed manually.
        • Also check that your Pro install went correctly. Check that you have a file edit_res.cgi under /usr/share/webmin/virtual-server/pro directory.
        • Ok, thanks! The last part revealed I didn’t complete the upgrade process. I thought it would be enough just to add the licence via terminal. Googled a bit more and found that I had to run the upgrade process via Virtualmin admin!
    • Does Virtualmin have limit CPU cores or CPU percent and RAM usage for each Account Plans? - Virtualmin - Virtualmin Community
      • khanhpkvn
        • Q: I have the (Virtualmin --> Manage Virtual Server --> Edit Resource Limits) menu. But "Edit Resources Limits" does not have CPU Core limits, it only has CPU "Number of processes". I want to able to limit CPU Cores/CPU Percent Usages and RAM per Account Plans.
        • A:
          • We use pam_limits (limits.conf) for these features, which do not have that sort of capability. cgroups can do it (sort of, though it also doesn’t think of CPU limits the way humans do), but we don’t yet have that support in Virtualmin. It’s on the todo list, but for now, there are a few ways you can have applied equally to all users, or based on a secondary group.
          • If you want all domains to have the same limits, the templates example at the bottom of this would be pretty quick to implement:
          • If they need to be different and selected at creation time or when moving from one account plan to another, it’d take either a little bit of scripting in a Server Templates post-update script, or just adding a secondary group (in Administrative user->Add domain owners to secondary group) and then setting up a group for each size, would work, I think. Since Account Plans can select the Server Template to use, this could make it all handled via choosing an Account Plan.
          • At least, I think setting it up with one group per “size” would work. I haven’t tried and the docs aren’t clear if every user in the group shares the same group limit or if they each get their own pool of resources. I need to read up some more, as I think we’d like to try to get it supported by Virtualmin 8, now that all of our supported distros have systemd (which, realistically, is required for cgroups support…theoretically one could use cgroups without it, but it’d be complicated to DIY a solution, I think).
    • What happens when updating from GPL to Pro to a server with ~30 domains? Pro has limit to 10 domains. - Virtualmin - Virtualmin Community
      • yngens
        • I just need to limit CPU and RAM consumption for couple of too much aggressive virtual servers
      • Eric
        • Ah, you can actually do all that on a system running Virtualmin GPL – you would just need to update the config file manually.
        • The settings that Virtualmin Pro edits for CPU and RAM usage are located in /etc/security/limits.conf.
        • Virtualmin Pro offers a GUI which allows you to set the cpu, rss, and nproc parameters… though there’s a number of additional parameters in there that you can tweak.
        • You can see some examples at the top of the limits.conf file, and some additional examples by running “man limits.conf”.
        • The CPU and RAM limits you can set are per-process though, and not per Virtual Server.
      • yngens
        • if I am not mistaken ‘/etc/security/limits.conf’ regulates resource usage time, not their power. I can’t set, for example, 20% of CPU and RAM consumption per virtual server.
      • Eric
        • No, those settings don’t allow you to set a specific percentage of the CPU or RAM that can be used for a given Virtual Server.
        • Those parameters each affect one specific process – so you can say how large a process a given user may create, or how much CPU time that process can use.
        • But, you can’t say “User N can use 20% of the CPU”.
        • Normally to achieve that sort of control, you’d look into separating a given user’s processes into a VPS, which can be more easily managed.
        • That said – there’s a new Linux kernel feature called cgroups which allows more functionality in that regard… it may be worth exploring the use of that in Virtualmin. It would take some time before that feature were supported, but we should probably take a look at that and see if it might be relevant for solving this particular problem :slight_smile:
        • However, you could always look into setting that up manually in the meantime. You can read about cgroups here: cgroups - Wikipedia

Importing from cPanel

  • Migration from cPanel to Webmin/Virtualmin - Interserver Tips - Virtualmin can import the accounts from cPanel by taking the complete cPanel backup file including all mailboxes, databases, contents,.. etc. This kind of migration process is much faster than others but need a special attention because some of the features of Webmin is not enabled automatically when you migrate the site. The site will work after migration but need to enable special features that only specified by the Webmin with care and testing. To copy or transfer all the services from cPanel to Virtualmin, first of all we need to take the fresh backup of them. We can generate the full cPanel backup by using the following steps:

Services (Daemons)

General

Apache (HTTP)

  • General
  • HTTP/2
    • HTTP/2 is enabled by default in Virtualmin
    • The HTTP protocols are: 
      Defined here: /etc/apache2/mods-available/http2.conf
Defined as: Protocols h2 h2c http/1.1
    • Webmin --> Servers --> Apache Webserver --> Global configurations --> Configure Apache Modules --> http2 = enabled
    • Virtualmin --> Web Configuration --> Website Options --> Enable HTTP2 protocol support = Default (Yes)
    • Virtualmin, Webmin and Usermin do not run under Apache or Nginx, They use miniserv.pl and this does not have HTTP/2 support.
  • Common Errors
    • Ubuntu default holding page is shown
      • Fix
        • Complete the Virtualmin setup process
        • Create at least one virtual server in Virtualmin.

    • 403 Forbidden
      • Fix = Create an index.html or other viable index file.

    • Virtualmin Holding page is shown
      • Fix = Add some content into the virtual sever.
    • 503 Service Unavailable
      • A Scenario
        • When I created a new virtual server (example.com) and then a sub-server (testest.example.com), this broken some of my other sites and they gave me the 503 error.
      • Solution
        1. Webmin --> System --> Bootup and Shutdown (Systemd)
        2. Make sure the relevant PHP services are set to 'Start at boot'.
        3. Restart the affected PHP services even if they say they are running.
        4. If the above does not work, consider rebooting the whole server.
      • Links
        • Apache 503 error - Here's how we nailed it
          • Apache 503 error means the server was temporarily unable to handle the website request. Service becomes unavailable due to wrong Apache, PHP settings.
          • Includes diagnostic steps.
        • Website gives 503 error when VPS is restarted - #8 by tpnsolutions - Help! (Home for newbies) - Virtualmin Community
          • Are you running multiple versions of PHP?
          • If so, it might be a different version of PHP-FPM that you need to restart.
          • The issue was simply this: both services related to FPM (php-fpm.service and rh-php72-php-fpm.service) were not enabled by default on systemd. So I have simply enabled them and now I can reboot the VPS without any problems.
          • Even if the service appears to be “up” it looks like it’s become defunct.
          • Restarting the php-fpm does nothing, only saving php options (without changing anything) solves the problem. So there is something else on the saving php script that does the trick, but I don’t know what.
  • 'Options +FollowSymlinks' causes 500 error
    • Don't enable FollowSymlinks
      • This is insecure in shared hosting.
      • SymLinksIfOwnerMatch is more secure and does the same as FollowSymlinks but also checks the owner's permissions.
      • FollowSymlinks will cause 500 errors in Virtualmin because the default apache directives disable overriding this setting via htaccess files.
      • followsymlinks on apache why is it a security risk - Server Fault
      • Server templates not properly applied · Issue #749 · virtualmin/virtualmin-gpl · GitHub
        • So the change from FollowSymLinks to SymLinksIfOwnerMatch is intentional as a security measure - otherwise, the owner of one domain could create a symlink to files in another domain's directory that are not normally accessible via the web, and make them accessible. The owners check prevents this.
      • Joomla 3.0 htaccess: Options +FollowSymLinks
        • FollowSymLinks is a vulnerability by itself on shared hostings, as it does NOT check for owners and thus allows customers to access any part of the system, including other accounts on the same server. It thus is / should be disabled by now on most hosting panels.
        • The new Apache2 directive to use is: Options +SymLinksIfOwnerMatch
    • Joomla
      • If this is enabled in your Joomla's .htaccess file, on Virtualmin, this will stop your website working, so change your file as follows: 
        Options +FollowSymlinks

-->

#Options +FollowSymlinks
Options +SymLinksIfOwnerMatch
    • General
  • Symlink directives - location and purpose
    • The Apache directory options are controlled in the Virtualmin GUI here:

      • Virtualmin --> Web configuration --> Configure Website / Configure SSL Website --> Document Options
      • Webmin --> Servers --> Apache Webserver --> Virtual Server --> Document Options
    • Webmin --> Servers --> Apache Webserver --> Global configuration --> Edit Config Files
      • This directive file is loaded by all virtual host before their specific directive file.
      • The 'AllowOverride None' directive disables the use of .htaccess files in this directory,
      • This is the 'Directives For default server' 
        # Sets the default security model of the Apache2 HTTPD server. It does
# not allow access to the root filesystem outside of /usr/share and /var/www.
# The former is used by web applications packaged in Debian,
# the latter may be used for local directories served by the web server. If
# your system is serving content from a sub-directory in /srv you must allow
# access here, or in any related virtual host.
<Directory />
    Options FollowSymLinks
    AllowOverride None
    Require all denied
</Directory>

<Directory /usr/share>
    AllowOverride None
    Require all granted
</Directory>

<Directory /var/www/>  -- this might be to allow virtualmin to work with allowing clients to use this
    Options Indexes FollowSymLinks
    AllowOverride None
    Require all granted
</Directory>

#<Directory /srv/>
#	Options Indexes FollowSymLinks
#	AllowOverride None
#	Require all granted
#</Directory>
    • Webmin --> Servers --> Apache Webserver --> Existing virtual hosts --> Type: 'Default Server' --> Show Directives
      • The server configuration by default has 'FollowSymLinks' disabled and cannot be overridden in a htaccess
      • Edit the 'Directives For default server' and you will see 
        <Directory />
 Options FollowSymLinks                                    /etc/apache2/apache2.conf (160)
 AllowOverride None                                        /etc/apache2/apache2.conf (161)
 Require all denied                                        /etc/apache2/apache2.conf (162)
</Directory>
    • Virtualmin --> pick a domain --> Web Configuration --> Configure Website / Configure SSL Website --> Edit Directives
      • This directive allows SymLinksIfOwnerMatch and is read after the default apache directives. 
        <Directory /home/example/public_html>
    Options -Indexes +IncludesNOEXEC +SymLinksIfOwnerMatch 
    Require all granted
    AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch
</Directory>

Nginx

I have not used this.

ProFTPd (FTP)

  • General
    • FTPeS, FTPS, Explicit FTP over SSL/TLS - General Discussion - Virtualmin Community
      • Describes where to enable 'FTP over SSL/TLS' in many different softwares.
      • Gives a list of the different names that have been assigned to 'FTP over SSL/TLS'.
    • Security Questions – Virtualmin
      • How can I prevent FTP Users from Browsing the Entire Filesystem?
        • If you want to limit the ease of which an FTP user can browse the server, you can setup FTP directory restrictions in Limits and Validation -> FTP Directory Restrictions. That would allow you to lock an FTP user into their home directory.
        • Note that this only prevents an FTP user from browsing the system, there are other ways in which a user can do the same thing.
        • Virtualmin --> Limits and Validation -> FTP Directory Restrictions
      • How can I prevent other types of users from browsing the entire filesystem?
        • On Linux/UNIX-based systems, users can browse to any file or directory they have permission to view.
        • That means any file or directory setup as world readable is visible to your users. In general, this is not a problem. The private data of other users is not something your users can browse by default.
        • Linux and UNIX systems weren't designed to act as jails, completely hiding one user from another.
        • Files that aren't okay for your users to see aren't made world readable.
        • Even if you were to jail an FTP user into their home directory, a web-based file manager would allow that user to browse world readable files on your server, since they still have permission to access them.
      • I just setup my server, and installed Virtualmin. Are there any steps I can take to improve the server security?
    • ProFTPD: FTP and SSL/TLS | proftpd.org - Config examples for TLS on ProFTPd.
    • ProFTPD: Logins and Authentication | proftpd.org - Logging into proftpd and being successfully authenticated by the server involves a lot of different modules and different checks. This document aims to discuss the sort of checks and configuration involved, and hopefully provide a better idea of how proftpd authenticates users. 
    • ProFTPD Core Module Documentation | proftpd.org - The following is a collection of HTML documentation for modules and tools in the modules/ directory of the ProFTPD source distribution.
    • The ProFTPD Project: Project Documentation | proftpd.org
      • The Official ProFTPD documentation.
      • The core documentation is held on GitHub, if you have patches or changes please submit them as a diff against those sources where possible. All updates should be directed to ProFTPD Core and they will be dealt with as quickly as possible.
  • FTP user home directory restrictions
    • By default Owner's accounts are restricted to the root of their home directory, but this can be changed by updating a permission.
      • Virtualmin --> System Settings --> Virtualmin Configuration --> Configuration category: Server administrator permissions --> Can select home directories for users
      • If enabled, the domain owner can choose to give users a different home directory than the default. It may be useful if domain owners can install additional services, like an application server (i.e. Zope, Webrick, etc.), and they'd like to be able to use a different user for the app server. Generally, only users that have a relatively high level of trustworthiness will need this kind of flexibility.
    • Secondary FTP users can either be locked to the owner's home root directory or to a specified subdirectory.
      • Virtualmin --> Edit Users --> Add a website FTP access user --> Quota and home directory settings --> Home directory:
    • Control home directories, directly with ProFTPd
      • Webmin --> Servers --> ProFTPd Server --> Files and Directories --> Limit users to directories:
      • I don't know how this differs from the options above, but here it is anyway. Perhaps if I made changes with the above options then those would be refelcted on this page.
  • Connection Issue - Status: Server sent passive reply with unroutable address. Using server address instead.
  • Denied FTP Users List
    • If the user's name is found in this file, FTP access is denied.
    • Enable/Disable this feature, by default this list is used
      • Webmin --> Servers --> ProFTPD Server --> Authentication --> Deny users in /etc/ftpusers file?
        • Default: Default (= On)
    • The list
      • Webmin --> Servers --> ProFTPD Server --> Denied FTP Users
    • UseFtpUsers - ProFTPD module mod_auth | proftpd.org
  • Allow login by root
    • Webmin --> Servers --> ProFTPD Server --> Authentication --> Allow login by root?
      • Default: Default (= Off)
    • Webmin --> Servers --> ProFTPD Server --> Denied FTP Users
      • Remove `root` from the list
    • Webmin --> Servers --> ProFTPD Server --> Apply Changes (This will restart ProFTPD).
    • RootLogin - ProFTPD module mod_auth | proftpd.org
    • Enable Login as Root in ProFTPD | Morz Project - WARNING, login as root is always bad practise. This tutorial intended for special case and running from within development environment.

PHP

MultiPHP
  • Installing additional versions (Virtualmin)
    • When you install a newer version of PHP-CLI this will change the system default PHP to this new version and you will need to manually change it back using a command such as update-alternatives if required.
    • When you remove the system default PHP-CLI, the highest remaining PHP version will become the new system default.
    • Multiple PHP Versions – Virtualmin
      • Managing and installing multiple PHP versions.
      • Adding another PHP version is outlined on this page. 
        LC_ALL=C.UTF-8 add-apt-repository -y ppa:ondrej/php && apt-get update

apt-get install php8.1-{cgi,cli,fpm,pdo,gd,mbstring,mysqlnd,opcache,curl,xml,zip}
    • My Upgraded commands (use these)
      • Install additional PHP versions will all the required modules
        ## Suitable for Joomla, WordPress (Required + Highly Recommended + Fallback + Cache) and General Hosting
apt-get install php8.1-{cli,fpm,opcache,common,bcmath,bz2,curl,gd,imagick,imap,intl,ldap,mbstring,mysql,readline,soap,tidy,xml,xmlrpc,zip}
      • Install all of my recommended PHP modules to all of the installed versions of PHP, this is good to make sure all the versions have the same modules. 
        for php in $(ls /etc/php); do sudo apt-get install -y "php$php-"{cli,fpm,opcache,common,bcmath,bz2,curl,gd,imagick,imap,intl,ldap,mbstring,mysql,readline,soap,tidy,xml,xmlrpc,zip}; done
    • PHP Interpreters
      • php8.1-cli
        • Command interpreter, useful for testing PHP scripts from a shell or performing general shell scripting tasks
        • If you want to run PHP from the terminal or SSH, then this is needed.
        • This should be install unless there is a specific reason not to
      • php8.1-cgi (not currently in my list)
        • Common Gateway Interface
        • By default, PHP is built as both a CLI and CGI program, which can be used for CGI processing. If you are running a web server that PHP has module support for, you should generally go for that solution for performance reasons. However, the CGI version enables users to run different PHP-enabled pages under different user-ids.
        • This is the slowest mode to run PHP in.
        • Allows CGI apps to run in the cgi-bin folder.
        • This is a legacy service and should not be installed unless you need it.
      • php8.1-fpm
        • FastCGI Process Manager, optimizing request handling.
        • This will install both PHP-FPM and FastCGI.
          • FastCGI
            • Currently this will not run FastCGI apps because this feature is not configured
          • PHP-FPM
            • This is the fastest mode to run PHP
            • This is the recommend standard for running PHP.
            • Will not run CGI apps.
    • Extensions Explained
      • Cache and Common
        • php8.1-opcache - Caches precompiled script bytecode to boost PHP performance.
        • php8.1-common - Offers functionalities common to various PHP modules / Documentation, examples, and common modules for PHP
      • Regular Extensions
        • php8.1-bcmath - Handles precise floating-point arithmetic and is used when working with precision floats
        • php8.1-bz2 - bzip2 module for PHP
        • php8.1-curl - lets you make HTTP requests in PHP
        • php8.1-gd - Image manipulation library for working with images
        • php8.1-igbinary (not currently in my list)
          • Igbinary is a drop in replacement for the standard PHP serializer.
          • Instead of the time and space consuming textual representation used by PHP's serialize(), igbinary stores PHP data structures in a compact binary form. Memory savings are significant when using memcached, APCu, or similar memory based storages for serialized data. The typical reduction in storage requirements are around 50%. The exact percentage depends on the data.
          • Perhaps only use this is you have enought resources and a full cache system in place.
          • This is rcommend by the WordPress requirements page.
          • I have not seen any hosting companies use this.
          • GitHub - igbinary/igbinary
        • php8.1-imagick - Image processing with ImageMagick.
        • php8.1-imap - These functions enable you to operate with the IMAP protocol, as well as the NNTP, POP3 and local mailbox access methods.
        • php8.1-intl - Supports international character sets.
        • php8.1-ldap - LDAP module for PHP
        • php8.1-mbstring - used to manage non-ASCII strings / Manages multibyte character encodings.
        • php8.1-mysql - Provides APIsfor working with MySQL databases
        • php8.1-pspell (not currently in my list)
          • These functions allow you to check the spelling of a word and offer suggestions.
          • TinyMCE | spellchecker - uses this for spell checking
          • Pspell extension moved from PHP Core to PECL - PHP 8.4 • PHP.Watch
            • The Pspell extension provides spell-checking features to PHP using Pspell or Aspell. The dependencies of this extension have not received any updates for the past few years, and the Pspell extension was moved away from PHP core to a PECL extension in PHP 8.4.
            • The Enchant extension (part of PHP core) is another extension providing spell-checking functionality to PHP. Unlike Pspell which only supported Pspell and Aspell, Enchant provides support for a wide list of backends including Hunspell and Ispell as well as Pspell/Aspell. The Enchant extension is not a direct drop-in replacement for Pspell extension functionality.
        • php8.1-readline - Facilitates interactive terminal input.
        • php8.1-snmp (not currently in my list)
          • The SNMP extension provides a very simple and easily usable toolset for managing remote devices via the Simple Network Management Protocol.
          • Only found this on cPanel servers so it might be a very niche usage.
        • php8.1-soap - The SOAP extension can be used to write SOAP Servers and Clients. It supports subsets of » SOAP 1.1, » SOAP 1.2 and » WSDL 1.1 specifications.
        • php8.1-tidy - Tidy is a binding for the Tidy HTML clean and repair utility which allows you to not only clean and otherwise manipulate HTML, XHTML, and XML documents, but also traverse the document tree, including ones with embedded scripting languages such as PHP or ASP within them using object-oriented constructs.
        • php8.1-xml - For XML parsing and manipulation. / DOM, SimpleXML, XML, and XSL module for PHP
          • Also provides: dom,SimpleXML,xmlreader,xmlwiter,xsl
        • php8.1-xmlrpc
          • Provides XML-RPC server and client functions.
          • http://xmlrpc.com/
          • What is XML-RPC? - It's a spec and a set of implementations that allow software running on disparate operating systems, running in different environments to make procedure calls over the Internet.
        • php8.1-zip - Manages zip file operations and for working with compressed files.
  • Install Command Build Notes
    • PHP: Extension List/Categorization - Manual
      • Full list of official PHP extensions.
      • This appendix categorizes more than 150 extensions documented in the PHP Manual by several criteria.
    • cgi,cli,fpm
      • These might not all be required and you should remove them as required.
      • Add notes from forum post
    • Some modules are built into the PHP binary at compile time such as: json,openssl,pcre,zlib and a few others, which is why there are not in the list above. Different version of PHP might have slightly different inbuilt modules.
      • i.e json used to be a separate modul,e but is now required to be added in at compile time.
    • Linux Packages can contain multiple PHP Extensions and PHP Extensions can contain multiple modules.
    • CMS Requirements
      • Server Environment – Make WordPress Hosting - Although WordPress can work in almost any environment, even very minimal ones, it must be acknowledged that it does not work completely well in these. That’s why here we are going to make some minimum recommendations of the environment in which it would work most effectively when considering that most WordPress websites use third party plugins and themes which commonly introduce additional server-level requirements.
      • J4.x:Optional Technical Requirements - Joomla! Documentation - This page lists optional technical requirements which are not required to install and run Joomla! but are required for some internal APIs.
      • J5.x:Optional Technical Requirements - Joomla! Documentation - This page lists out optional technical requirements which aren't required to actually install and run Joomla! but are required for some dependencies running different internal APIs.
    • How to install or upgrade to PHP 8.3 on Ubuntu and Debian • PHP.Watch
      • A complete guide to install or upgrade to PHP 8.3 on Ubuntu 22.04 (Jammy), Ubuntu 20.04 (Focal), Debian 10 (Buster), 11 (Bullseye), and Debian 12 (Bookworm).
      • The php8.3-common package is a meta-package that installs several PHP extensions. It is possible to selectively disable individual extensions later. PHP Core extensions such as Date, Phar, JSON, ctype, and random are always included. It is not necessary nor possible to install them as separate packages.
      • Instead of installing php8.3-common, it is also possible to install individual packages. Installing php8.3-common is roughly equivalent to installing all of the extensions as shown below: 
        apt install php8.3-{calendar,ctype,exif,ffi,fileinfo,ftp,gettext,iconv,pdo,phar,posix,shmop,sockets,sysvmsg,sysvsem,sysvshm,tokenizer}
      • php8.3-cli installs the PHP CLI, and symlinks /usr/bin/php to /usr/bin/php8.3. See Running PHP 8.3 Alongside Other Versions for more information.
    • Required PHP Extensions For WordPress - WPQuickies - In this lunchtime WPQuickies, I'll be listing the required PHP extensions that WordPress needs to run properly.
    • WordPress Required PHP Extensions // WPAssist - WordPress needs PHP extensions to generate page content, update core and plugins and also for handling of file and image uploads. In this post, we have compiled a complete list of required PHP extensions for operating WordPress on a linux server.
    • How To Install PHP 8.1 and Set Up a Local Development Environment on Ubuntu 22.04 | DigitalOcean
      • This tutorial will guide you through installing PHP 8.1 on Ubuntu and setting up a local programming environment via the command line.
      • Gives a list of extensions to install.
    • How to Install PHP 8.3 on Ubuntu 22.04 or 20.04 - LinuxCapable
      • Commands to install PHP 8.3 on Ubuntu 22.04 or 20.04 LTS from a well-known PPA. Includes PHP 8.3 Apache, Nginx and Modules tips.
      • Gives a list of extensions to install.
    • Our PHP Modules | Hostgator - This article contains lists of the PHP modules and PEAR packages pre-installed on our server, as well as the basics for configuring the TimThumb script. Discover them all now!
    • Complete Guide on How to Install PHP Extensions on Ubuntu Complete Guide on How to Install PHP Extensions on Ubuntu - Ubuntu is a Linux distribution that is popular for web development, server hosting, and other applications. PHP is a server-side scripting language that is widely used for web development and is extensively supported on Ubuntu. In this article, we will discuss what are PHP extensions, and the advantages of installing them on Ubuntu. We’ll also discuss the steps on how to install those extensions on Ubuntu.
  • The Installation Process
    • Some of the specified packages will get installed as dependecies of other packages, however it does not harm having then in the list as it also makes it easy for the installer to know what is going to get installed. 
      root@example:~# apt-get install php7.1-{cgi,cli,fpm}

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  php7.1-common php7.1-json php7.1-opcache php7.1-readline
The following NEW packages will be installed:
  php7.1-cgi php7.1-cli php7.1-common php7.1-fpm php7.1-json php7.1-opcache php7.1-readline
0 upgraded, 7 newly installed, 0 to remove and 1 not upgraded.
Need to get 5270 kB of archives.
After this operation, 23.8 MB of additional disk space will be used.
Do you want to continue? [Y/n]
    • If the package has been almagamated into another package, you usually find it is now a virtual package (alias) to allow for compatability. 
      php-mysqlnd --> php8.1-mysqlnd --> php8.1-mysql
php-pdo --> php8.1-pdo --> php8.1-common
    • If you have specified a package to install that is already installed, the installer will just skip it.
    • If you have specified a package that has just been installed as part of an earlier package during the install, the installer will just skip it.
    • All the default PHP packages are aliases to the real version as shown below: 
      php-cgi --> php8.1-cgi
php-common --> php8.1-common
php-gd --> php8.1-gd
    • The command above cycles through the commands within {} and combines them with php8.1- to give for example php8.1-cgi and then these commands runs one after another.
    • After a package is installed, Ubuntu will keep a record of where it was downloaded from.
    • Not all PHP extensions have binary files you can download, they need to be added at compile time.
      • eg OpenSSL: This is compiled when you build PHP rather than it being a 'Dynamic Extension'.
    • When installing packages most the time there is a one to one relationship between them and the php extension you expect to be installed, this is not always the case. One package can install multiple extensions and also bring in other extensions via dependencies.
    • If you install a newer version of CLI PHP than the system default PHP, then the system default will be changed to this new version of PHP.
    • If you uninstall the latest version of PHP and this happens to be set as the system default PHP, the high remaining PHP version will become the system default verison.
    • This assumes it has the CLI version of PHP installed.
  • Ondrej Repository
    • The ondrej repository takes priority over the standard repositories. This will be checked for updates first.
    • This only hold PHP packages to allow the installation of additional PHP versions.
    • Most likely all of your PHP updates will no come from this repository, including for the system default PHP version.
    • Site Links
  • Installing additional versions and changing the system default php version
  • Remove old version of PHP
  • Changing a Virtual Server's PHP version
    • You must have multiple version of PHP to be installed for this to work.
    • You can configure the PHP version being used for a specific Virtual Server by selecting:
      • Virtualmin --> Web Configuration --> PHP Options.
    • What happens when a user swaps their PHP version?
      • The settings configured via the GUI are maintained between PHP versions. So Virtualmin must edit the config files as required when the version is changed.
      • Virtualmin --> Web Configuration --> PHP-FPM Configuration
    • Virtualmin - Install PHP 8.0 and update all sites - Dennis Tsang
      • This blog post outlines the steps of installing and configuring PHP 8 on an existing install of Virtualmin on a Ubuntu system
      • Then you can update all the Virtualmin sites to use the new version with this API command: 
        virtualmin modify-web --all-domains --mode fpm --php-version 8.0
PHP Information
  • Show PHP Version
  • PHP Module Config Files 
    /etc/php/
/etc/php/8.1/
/etc/php/8.1/cgi/
/etc/php/8.1/cli/
/etc/php/8.1/fpm/
/etc/php/8.1/mods-available/
    • You can look in the /mods-available/ folder to see what has been installed. This might not give the same results as php -m but should be close, if not the same.
  • Show Package Information
    • How to Check Dependencies of a Package in Ubuntu/Debian-based Linux Distributions - Installing applications via command line is quite easy in Ubuntu/Debian. All you need to do is to use apt install package_name.But what if you want to know the dependencies of a package before or after installing it? In this tutorial, I’ll show you various ways to see the dependencies of a package in Ubuntu and other Debian-based Linux distributions that use APT package management system. 
      apt show php8.1-fpm      - Gets info from Ondjre
apt show php8.1-fpm -a   - Gets info from Jammy repo
  • Check if a PHP Module is installed
    • Run one of these commands from the terminal to check if the particular PHP extension is available. You will get a result if the relevant extension is available. 
      ### System Default PHP Version

# Show compiled in modules
php -m

# Check for MySQLi
php -m | grep mysqli

# Check for MySQL PDO
php -m | grep pdo_mysql

# Check for MySQL Native Driver (mysqlnd)
php -m | grep mysqlnd
      ### Alternative PHP versions (installed from the Ondrej's PPA Repository)

# Show compiled in modules
/usr/sbin/php-fpm7.4 -m

# Check for MySQL MySQLi
/usr/sbin/php-fpm7.4 -m | grep mysqli

# Check for MySQL PDO
/usr/sbin/php-fpm7.4 -m | grep pdo_mysql

# Check for MySQL Native Driver (mysqlnd)
/usr/sbin/php-fpm7.4 -m | grep mysqlnd
    • PHP: Options - Manual | php.net
      • php -m = Show compiled in modules
    • How to List Compiled and Installed PHP Modules in Linux - If you have installed a number of PHP extensions or modules on your Linux system and you trying to list installed PHP extensions on your Linux system.
    • How to List Compiled PHP Modules from Command Line | Liquid Web
      • Want to know which PHP modules are installed on your server? Check out our tutorial on how to list compiled PHP modules from command line.
      • Covers cPanel.
    • The way I figured out where the binary was as follows
      • Webmin --> System --> Software Packages --> Search for Package: php7.4
      • Clicked on php7.4-fpm 7.4.33-8+ubuntu22.04.1+deb.sury.org+1
        • I wanted to know where the FPM binary was
      • List Files
        • This now shows all linked files
      • Sort by Type
      • Find the largest Regular File.
        • This will most likely be the binary file you want.
      • Get the the file path from this record and use it in the commands above as shown.
Global php.ini
  • Editing the global php.ini / PHP Configuration (cli/fpm/cgi)
  • Global php.ini - how they are used
    • Based on my research and feedback from Use the global php.ini for my sites instead of the copied one - Virtualmin - Virtualmin Community this is how the global php.ini files behave
      • default php (/etc/php.ini)
        • There is not GUI mechanism for editing this.
        • Unsure how this behaves.
      • cli
        • Unsure how this behaves.
      • cgi
        • These are only use as templates for your 'CGI wrapper’/FCGId PHP based Virtual Servers at the point of creation.
        • These files are copied as outlined below.
      • fpm
        • These are used by all Virtual Servers running PHP-FPM and clients can then override the values on a per Virtual Server basis if their permissions allow them.
    • DNS options - PHP Template configuration files (explained)

      This mechanism/feature does not affect PHP-FPM as that uses the global fpm php.ini and then uses a per Virtual Server override system using .conf files. I am not sure if .user.ini files will allow per folder overrides.

      • The php.ini Copy Mechanism
        • When you create a Virtual server, the 'CGI wrapper’/FCGId PHP global php.ini for each of the different versions installed (eg: 7.4, 8.1, 8.2) are copied in the following way to the virtual servers home directory. You will also note that symlink has been created to your servers php.ini file for the default installed version of PHP.
          # Copied Files
/etc/php/7.4/cgi/php.ini --> /root/home/example/etc/php.7.4
/etc/php/8.1/cgi/php.ini --> /root/home/example/etc/php.8.1
/etc/php/8.2/cgi/php.ini --> /root/home/example/etc/php.8.2

# Symlink
/root/home/example/etc/php.ini --> /etc/php.ini
        • These default destination of the files can be changed by using an option in the 'Server Template' (only works at creation of the Virtual Server).
          • Virtualmin --> System Settings --> Server Templates --> template --> Edit template section: Php options --> Template PHP x.x configuration file.

          • The tool tip says: By default, when a virtual server runs PHP scripts as the server's owner, Virtualmin will copy the system's global PHP configuration file (usually/etc/php.ini) to the domain's ~/etc directory. This allows PHP options to be set differently on a per-server basis.
      • The Why
        • The copied php.ini files are used for ‘CGI wrapper’/FCGId and they were/are a way of Virtualmin allowing each server to have their own php.ini settings.
        • This is useful:
          • If you wish to serve different versions of php in different directories of the same domain,
          • I believe that in future versions of the virtualmin module that you will be able to use FPM to achieve the same goal,
          • but just be aware in the current version of the virtualmin module, Virtualmin writes a symlink to ~/etc/php.ini which is linked to the version of php you have chosen (e.g 8.1), which in turn messes with the cli installation of php for that user.
          • For example you may want to serve the web pages using php 7.x, but allow any exec’s from the web content to use php 8.x, it will not, as it uses the version of php.ini that the symlink points to (in this case 7.x). To get around the problem delete the symlink.
        • This mechanism is used instead of just overriding the global php.ini with individual values like how cPanel does it.
        • This is definately geared more towards sysadmins and app developers.
        • This means, editing the CGI global php.ini files in `Webmin --> Tools --> PHP Configuration` is pointless as these php.ini files are just being used like Server Templates, a copy is made and that is used but then the copy is nver updated again by the system.
      • The Bad
        • As you role out new Virtual servers, their php.ini will become out of sync with the global and this is a bad way of managing servers for webhosting. I like to know what they are all set at and I can change them to be all the same.
        • The current setup will just have servers on different snapshots of the global php.ini at different times even if they are not touched.
    • Overview
      • Just use PHP-FPM
        • It allows one central php.ini that can be overridden by clients when required, if their permissions allow them.
        • sysadmins can push update 'disable_fuctions' quickly and easily to all clients on a particular PHP version.
        • PHP-FPM is much quicker that the other versions of PHP, and possible more secure.
Binary Locations
/usr/bin/php
/usr/bin/php7.4
/usr/bin/php8.3
/usr/bin/php-cgi
/usr/bin/php-cgi7.4
/usr/bin/php-cgi8.3
/usr/sbin/php-fpm7.4
/usr/sbin/php-fpm8.3
  • These are useful if you need to run ommand on specific PHP version or just get the relevant information
  • The system default PHP is just a symlink to an installed version of PHP, which you can change.
  • There is no default version of PHP-FPM
PHP-FPM
  • Better Settings / Performance
    • How To Prevent PHP-FPM From Consuming Too Much RAM in Linux - In this article, we will show how to prevent PHP-FPM from consuming too much or all your system memory (RAM) in Linux.
    • A better way to run PHP-FPM - If you search the web for PHP-FPM configurations, you’ll find many of the same configurations popping up. They nearly all use the ‘dynamic’ process manager and all assume you will have one master process for running PHP-FPM configurations. While there’s nothing technically wrong with that, there is a better way to run PHP-FPM.
    • How to Reduce Memory (RAM) Usage in PHP-FPM – TecAdmin - This article provides insights into several techniques that can help optimize the RAM usage of PHP-FPM and ensure smoother server operations.
Misc
  • General
    • How to upgrade Virtualmin scripts when PHP version checks fail · the.Zedt - When things have been running for long enough various updates and configuration changes start adding up with leftovers bound to cause an issue sooner or later. With Virtualmin, one such issue is the system's inability to automatically update its scripts to newer versions based on the incorrect detection that an older PHP version is running on the server instead of the actual one.

MariaDB (Database)

General
  • Misc
  • Add additional users toa database
  • Users cannot edit databases
    • This is usually caused because the 'Account Plan' used for the user was not configured with the correct permissions in the first place.
    • Fixes
      • Make sure the domain owner has the ability to edit databases and change as appropriate
        • Virtualmin --> Manage Virtual Server --> Edit Owner Limits --> Allowed capabilities and features --> Can manage databases
      • You can try swapping the 'Account Plan' to another and then back again after you have changed the 'Account Plan' permissions.
      • Make sure the database feature is enabled
        • Virtualmin --> System Settings -> Features and Plugins
  • How do i stop a single database being created during creating a virtual server?
    • System Settings -> Server Templates -> Template -> MariaDB Database --> Create database as well as login: No
  • Move a database between accounts
    1. Virtualmin (current owner) --> Edit Databases --> 'the database' --> Disassociate With Server
    2. Virtualmin (new owner) --> Edit Databases --> Import Database:
Creating SQL Databases with independant credentials on a single Virtual Server

There are 2 ways to create a database manually in virtualmin.

  1. Virtualmin --> Edit Databases --> Create a new database
    • When you create a database here, the database will belong to the Virtual Server owner, which is a good thing.
    • The Virtual Server owner's credentials will always give full access on these databases.
    • You should always use this option to create your databases so they are always owned by your Virtual Server owner's account and will ensure they are backed up with the rest of the account's files.
  2. Webmin --> Servers --> MariaDB Database Server --> Create a new database
    • When you create a database here, the database will belong to whoever you set it to.
    • Make sure you set it to the right owner so it is backed up with their files when an account backup is triggered.

There are a couple of different ways to add a additional SQL users to these databases

  1. Virtualmin --> Edit Users --> (Add a user to this server | Add a website FTP access user) --> Other user permissions --> Allow access to databases:
    • Creating a user here will allow you to configure an access  with a Virtualmin user, but will create other associated services along with it such as an email address, so is not ideal.
  2. Webmin (workaround)
    • Webmin --> Servers --> MariaDB Database Server --> User Permissions --> Create new user
      • This will give you full control over, Username, Password and what permissions this user can have because it is a native MariaDB SQL user.
      • Username: example_prestashop
      • Password: ********
      • Hosts: localhost
      • Permissions: none
        • These will be set below for the specified database.
        • These are global permissions. Only root and soime system accounts should have these.
      • Igore the rest of the settings
    • Webmin --> Servers --> MariaDB Database Server --> Database Permissions --> Create a new database permissions
      • This will allow you to connect your user to your database
      • Databases --> Selected: example_database
      • Username: example_prestashop
      • Hosts: localhost
      • Permissions: select all (or just those you require)
    • Webmin --> Servers --> MariaDB Database Server --> Database Permission
      • NB:
        • You will see that any underscores in your database name are escaped with a slash, this is normal behaviour because normally the underscore acts as a wildcard character.
        • If you use pattern matching, if you want to specify one database then you should always esape your underscores.

Notes

  • I have submitted a feature request to impreve this situation
  • 'Keep MariaDB and administration usernames in sync: Yes' = Is a good thing, this will allow you to login with your Virtual Servers username and the 'MariaDB database' password to phpMyAdmin and see all of your tables just like cPanel.
    • Virtualmin --> Edit databases --> Passwords --> MariaDB database
  • None of the Virtualmin options are a good choice for creating additional SQL users. The Webmin workaround will work but is not suitable for large numbers of clients, or for clients to use.
  • Is it possible to have a MySQL database owned by multiple Virtualmin owners? - #2 by leecf - Virtualmin - Virtualmin Community
    • You can also create Webmin users that have access any given database. Webmin’s MySQL module is incredibly powerful and flexible and has great ACLs.
    • To be clear: Database users and Webmin users are separate entities, but you can use either or both to provide access to any database, depending on what you’re trying to accomplish.
    • If you want web apps on different domains to share a database, you can create one or more MySQL database users in the MySQL module.
    • If you want to allow a user to manage another users databases in Webmin, you can create a new Webmin user just for that purpose. (Virtualmin users are kinda locked down to prevent their ACLs from being changed for safety…so we recommend a whole new user for sharing databases, but I think it’d be possible to make a database accessible to multiple Virtualmin domain owner users, if you click through the warning about it being a Virtualmin user).
    • MySQL Database Server | Webmin - On this page the MySQL database and the Webmin module managing it are explained, and the steps to follow to create databases, tables and users are listed.
  • How to set permissions for mysql - certain DBs ? - Help! (Home for newbies) - Virtualmin Community
    • All of this can be done using the Webmin MySQL module, though doing so does take it out of control of Virtualmin to some degree (Virtualmin loosely enforces a “virtual server --> databases” type of ownership hierarchy where you can have many databases, but each database has only one owner and it’s a virtual server owner account).
    • That said, I make use of the Webmin MySQL module extensively on Virtualmin.com to allow the existence of our development domains, independent access by our license manager, etc., and it’s not particularly dangerous to do so (it just means that some of the relationships and permissions are not obvious in the Virtualmin interface, since it doesn’t cover things that deeply).
    • So, to grant access to a database browse to Webmin:Servers:MySQL Database Server, and click on "Database Permissions". Here you can click "Create new database permissions." and build up fine-grained (or unlimited) access rules for any user to any database.

BIND (DNS)

  • Official
  • General
    • Set Up Local DNS Resolver on Ubuntu 22.04/20.04 with BIND9
      • This tutorial shows you how to set up a local DNS resolver on Ubuntu 22.04/20.04, with the widely-used BIND9 DNS software. Why Run Your Own Local DNS Resolver?
      • Usually, DNS queries are sent to UDP port 53. The TCP port 53 is for response sizes larger than 512 bytes.
      • The bind9 package on Ubuntu 22.04/20.04 doesn’t ship with a db.root file, it now uses the root hints file at /usr/share/dns/root.hints. The root hints file is used by DNS resolvers to query root DNS servers. There are 13 groups of root DNS servers, from a.root-servers.net to m.root-servers.net.
      • This is an excellent tutorial
  • DNS over HTTP (DoH)
  • DNS over TLS (DoT)
  • DNSSEC
    • Regenerate DNSSEC key for Virtual Server
      • Webmin --> Servers --> BIND DNS Server -> yourdomain -> Setup DNSSEC Key -> (Remove Key | Sign Zone | Re-Sign Zone)

systemd-resolved (DNS Resolver)

systemd-resolved is a system service that provides network name resolution to local applications. It implements a caching and validating DNS/DNSSEC stub resolver, as well as an LLMNR and MulticastDNS resolver and responder. Local applications may submit network name resolution requests via three interfaces:

Overview
  • systemd-resolved
    • is a systemd service that provides network name resolution to local applications.
      • is caching resolver
      • it has the same role as dnsmasq
      • is part of Ubuntu core and not Bind
      • is for apps and the command line to make DNS requests and uses /etc/resolve.conf for legacy apps.
      • is only bound on the loopback adapter at port 53 (i.e. 127.0.0.53:53, 127.0.0.1:53 etc…)
      • it provides the following
        • The native, fully-featured API systemd-resolved exposes on the bus.
        • The glibc getaddrinfo API as defined by RFC3493[1] and its related resolver functions, including gethostbyname.
        • A local DNS stub listener on IP address 127.0.0.53 on the local loopback interface (a.k.a. Stub Responder).
      • /etc/resolv.conf
        • is symlink and you should not edit this file as it will get regenerated upon `systemd-resolved`restart.
        • this is for legacy applications.
        • most CLI commands and apps seem to use this file for getting the nameservers when you don't specify them in the commain (i.e. @10.0.0.1)
      • The stub resolver gets nameservers from several places in cluding the network interfaces and /etc/resolv.conf if not a symlink.
  • bind
    • bind receives and responds to DNS requests on specified network interfaces, it does not send them.
  • 127.0.0.53
    • 127.0.0.1 seems the same as 127.0.0.53
    • 127.0.0.53 is only needed in your network card for the following reasons:
      • you have got DNS servers specified in /etc/systemd/resolved.conf
        and
      • you require SplitDNS
      • you want to use the benefits of the systemd-resolved DNS cache (for this one you would remove 127.0.0.53 from your network interface)
    • if 127.0.0.53 is specified in the network card I assume systemd-resolved ignores this to prevent an infinite loop
    • 127.0.0.53 is specified in my network card by default by Ubuntu when I set the server up.
    • 127.0.0.53 in your NIC is only any good if you have defined external DNS servers in /etc/systemd/resoveld.conf or ironically there is a real DNS server listed in your NIC.
    • I am assuming that `systemd-resolved` internally ignores 127.0.0.53 and 127.0.0.1 are nameservers.
  • Other Notes
    • 127.0.0.53 - Do I need it? - Virtualmin - Virtualmin Community
      • You’re correct, this is the default resolver configuration on modern systems. It isn’t weird, it isn’t unusual, it’s not a mystery. It’s usually systemd-resolved, as you note. That’s a caching resolver intended for local use.
      • Asking if you “need it” is a question only you can answer. You need it if you don’t want to do some work to change the way resolution works on your system.
      • There are many tools that can provide this service (local caching DNS resolution), and you don’t even really need caching local DNS on a server, in a lot of cases, since you’re not going to be doing a lot of time-sensitive DNS requests.
      • Since it’s not a desktop, most things that need DNS are not interactive…a few ms to go out to 8.8.8.8 or 1.1.1.1 probably isn’t going to be noticeable. (Though if you use a lot of APIs that are involved in interactive services, then you should have local caching DNS.)
General
  • GitHub - systemd/systemd: The systemd System and Service Manager
  • resolved.conf(5) — systemd-resolved — Debian bookworm — Debian Manpages
  • Ubuntu Manpage: systemd-resolved.service, systemd-resolved - Network Name Resolution manager
    • systemd-resolved is a system service that provides network name resolution to local applications. It implements a caching and validating DNS/DNSSEC stub resolver, as well as an LLMNR and MulticastDNS resolver and responder. Local applications may submit network name resolution requests via three interfaces:
      1. The native, fully-featured API systemd-resolved exposes on the bus,
      2. The glibc getaddrinfo API as defined by RFC3493[1] and its related resolver functions, including gethostbyname.
      3. A local DNS stub listener on IP address 127.0.0.53 on the local loopback interface (a.k.a. Stub Responder).
    • This resolver also implements LLMNR and MulticastDNS in addition to the classic unicast DNS protocol, and will resolve single-label names using LLMNR (when enabled) and names ending in ".local" using MulticastDNS (when enabled).
    • Four modes of handling /etc/resolv.conf (see resolv.conf(5)) are supported:
      1. systemd-resolved maintains the /run/systemd/resolve/stub-resolv.conf file for compatibility with traditional Linux programs. This file may be symlinked from /etc/resolv.conf. This file lists the 127.0.0.53 DNS stub (see above) as the only DNS server. It also contains a list of search domains that are in use by systemd-resolved. The list of search domains is always kept up-to-date.
      2. A static file /usr/lib/systemd/resolv.conf is provided that lists the 127.0.0.53 DNS stub (see above) as only DNS server. This file may be symlinked from /etc/resolv.conf in order to connect all local clients that bypass local DNS APIs to systemd-resolved. This file does not contain any search domains.
      3. systemd-resolved maintains the /run/systemd/resolve/resolv.conf file for compatibility with traditional Linux programs. This file may be symlinked from /etc/resolv.conf and is always kept up-to-date, containing information about all known DNS servers.
      4. Alternatively, /etc/resolv.conf may be managed by other packages, in which case systemd-resolved will read it for DNS configuration data. In this mode of operation systemd-resolved is consumer rather than provider of this configuration file.
  • systemd-resolved - ArchWiki
    • This says that the DNSSEC support is experimental, but this page has not been updated in a while and I don't believe it is experimental anymore
    • Test DNSSEC validation by querying a domain with a invalid signature: 
      Bad domain: resolvectl query badsig.go.dnscheck.tools
Good domain:  resolvectl query go.dnscheck.tools
  • RFC 4795: Link-local Multicast Name Resolution (LLMNR) | rfc-editor.org
  • Understanding systemd-resolved, Split DNS, and VPN Configuration – Michael Catanzaro's Blog
    • This is a very indepth article about `systemd-resolved` and you should pay attention to the `Servers and DNSSEC` section.
    • You might have noticed that the rest of this blog post focused pretty much exclusively on desktop use cases. Your server is probably not using a VPN. It’s probably not using mDNS. It’s probably not expected to be able to resolve local hostnames.
    • Conclusion: most servers don’t need split DNS! Servers do benefit from systemd-resolved’s systemwide DNS cache, so running systemd-resolved on servers is still a good idea. But it’s not nearly as important for servers as it is for desktops.
    • There are some disadvantages for servers as well. First, systemd-resolved is not intended to be used on DNS servers. If you’re running a DNS server, you’ll need to disable systemd-resolved before setting up BIND or Unbound instead.
  • Using a Specific DNS for a Specific Domain in Linux | Baeldung on Linux
    • Learn different ways to use specific DNS for certain domains or certain applications.
    • systemd-resolved is a systemd service that provides network name resolution to local applications.
    • Most major distributions now use systemd by default. So, chances are it’s already installed on our machine. We can check its status through the following command: 
      systemctl status systemd-resolved
  • linux - Why does /etc/resolv.conf point at 127.0.0.53? - Unix & Linux Stack Exchange
    • You are likely running systemd-resolved as a service.
    • systemd-resolved generates two configuration files on the fly, for optional use by DNS client libraries (such as the BIND DNS client library in C libraries):
    • /run/systemd/resolve/stub-resolv.conf tells DNS client libraries to send their queries to 127.0.0.53. This is where the systemd-resolved process listens for DNS queries, which it then forwards on.
    • /run/systemd/resolve/resolv.conf tells DNS client libraries to send their queries to IP addresses that systemd-resolved has obtained on the fly from its configuration files and DNS server information contained in DHCP leases. Effectively, this bypasses the systemd-resolved forwarding step, at the expense of also bypassing all of systemd-resolved's logic for making complex decisions about what to actually forward to, for any given transaction.
    • Much more relevant information here.
  • Example /etc/resolv.conf file 
    nameserver 127.0.0.53
options edns0 trust-ad
search .
DNSSEC General
Enable DNSSEC in `systemd-resolved`
  • Enable DNSSEC support in systemd-resolved - Stan's blog
    • Systemd-resolve is used in most systemd distributions. DNSSEC checking is disabled by default, so here is a quick tutorial to enable it.
    • Most of the recent systemd distributions use it, Ubuntu does since 16.10. It has the same role as dnsmasq.
  • DNSSEC for NetworkManager Using systemd-resolved · Felix Ehrenpfort
    • The default DNS backend used by NetworkManager doesn’t seem to support DNSSSEC. 
      dig www.dnssec-deployment.org | grep status
# ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59040

dig www.dnssec-failed.org | grep status
# ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 34764
    • DNSSEC for systemd-resolved is enabled by setting the DNSSEC field under the Resolve section inside /etc/systemd/resolved.conf to true. 
      DNSSEC=true
Disable systemd-resolved
Override Ubuntu DNS using systemd-resolved
Commands
## Stop / Start / Restart / Disable
service systemd-resolved restart  (old systems)
systemctl restart systemd-resolved
systemctl stop systemd-resolved
systemctl start systemd-resolved
systemctl disable systemd-resolved

## Status
systemd-resolve --status    (old systems)
systemctl status systemd-resolved                 = shows systemd-resolved service info
systemctl status                                  = shows all services runnning in a nice tree
resolvectl status                                 = shows the systemd-resolved status
resolvectl status | grep Protocols                = shows systemd-resolved supported protocols
resolvectl status | grep -i protocols             = shows systemd-resolved supported protocols(lookup is case-insensitive)
service --status-all                              = shows all installed services and their status
service systemd-resolved status                   = shows systemd-resolved service info

## Logging
resolvectl log-level                              = show log level
resolvectl log-level debug                        = set log level to debug

## Misc
resolvectl -h                                     = show resolvectl help
systemd-analyze cat-config systemd/resolved.conf  = show resolved config file
Config Files
# The standard file which is usually a symlink and is kept for legacy reasons. It usually points to stub-resolv.conf 
/etc/resolv.conf

# Main Config File
/etc/systemd/resolved.conf

# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
# (Dynamically created)
/run/systemd/resolve/stub-resolv.conf 

# (Dynamically created)
/run/systemd/resolve/resolv.conf

# Other
/etc/resolv.conf  (real file version)
/usr/lib/systemd/resolv.conf
  • /run/systemd/resolve/stub-resolv.conf tells DNS client libraries to send their queries to 127.0.0.53. This is where the systemd-resolved process listens for DNS queries, which it then forwards on.
  • /run/systemd/resolve/resolv.conf tells DNS client libraries to send their queries to IP addresses that systemd-resolved has obtained on the fly from its configuration files and DNS server information contained in DHCP leases. Effectively, this
Troubleshooting (systemd-resolved)
  1. resolvectl / sd_bus_open_system: No such file or directory
    • Background
    • Cause:
      • The dbus service is not running
    • Workaround:
      • Webmin --> System --> Bootup and Shutdown
      • restart the dbus service
      • This does not survive a server restart.
    • Fix:
      • Disable the dbus service from startup and then re-add it
        • Webmin --> System --> Bootup and Shutdown --> dbus: Disable On Boot
        • Webmin --> System --> Bootup and Shutdown --> dbus: Start On Boot
    • Successful


    • Notes
      • Currently I cannot re-enable dbus on startup with Webmin because of a bug, so I used the following command from the terminal.
        sudo systemctl enable dbus
      • Might need to install dbus
        • On some versions of Linux dbus might not be installed and thus would need to be. This should not be the case for full OS but maybe for Linux flavour for RaspberryPis etc..
        • systemd-resolve doesn't work - Troubleshooting - DietPi Community Forum 
          • ## by default DietPi is running with limited amount of packages. This include dbus package as well. Therefore it would need to be installed.

apt update
apt install dbus
systemctl enable dbus
reboot

##probably systemd-resolved service to be started as well

systemctl enable systemd-resolved.service --force
    • Resolvectl / sd_bus_open_system: No such file or directory - General Discussion - Virtualmin Community
    • Ubuntu: How to auto-start a service on system boot - Sling Academy - Introduction As a Linux administrator or user, you may need to ensure that certain services automatically start up every time your system boots.
  2. *.mail.protection.outlook.com not correctly resolved when DNSSEC is enabled.
    • Symptoms
      • Emails being deferred in your mail queue and ultimately not delivered becasue the domain cannot be resolved.

        ## AAAA
Host or domain name not found. Name service error for name=example-com.mail.protection.outlook.com type=AAAA: Host not found, try again

or

## A

Host or domain name not found. Name service error for name=example-com.mail.protection.outlook.com type=A: Host not found, try again
        • These errors means the DNS lookup is failing and you can see this happens for both IPv4 and IPv6.
        • If you disable IPv6 support in Postfix with the setting inet_protocols  and then reboot the server the lookups will all be retried with IPv4 and the DNS lookup will still fail.
      • Domains that do not use DNSSEC do not seem to be affected.
    • Diagnostics
      • You can test if the DNS results by using resolvectl, nslookup, ping, Delv and Dig to see the full results looking for SERVFAIL or NOERROR and then you can also force these commands to directly use you router's DNS server to bypass the Stub Resolver and compare results which will more than likely show the Stub Resolver is at fault. 
        ## Ping
theuser@server:~$ ping example-com.mail.protection.outlook.com
ping: example-com.mail.protection.outlook.com: Temporary failure in name resolution

## NSLookup
theuser@server:~$ nslookup
> example-com.mail.protection.outlook.com
;; Got SERVFAIL reply from 127.0.0.53
Server:         127.0.0.53
Address:        127.0.0.53#53

** server can't find example-com.mail.protection.outlook.com: SERVFAIL
> 

## Resolvectl
theuser@server:~$ resolvectl query example-com.mail.protection.outlook.com
example-com.mail.protection.outlook.com: resolve call failed: DNSSEC validation failed: failed-auxiliary

theuser@server:~$ resolvectl query mail.protection.outlook.com
mail.protection.outlook.com: resolve call failed: 'mail.protection.outlook.com' does not have any RR of the requested type

theuser@server:~$ resolvectl query protection.outlook.com
protection.outlook.com: resolve call failed: 'protection.outlook.com' does not have any RR of the requested type

theuser@server:~$ resolvectl query outlook.com
outlook.com: 52.96.222.226                     -- link: ens3
             52.96.214.50                      -- link: ens3
             52.96.229.242                     -- link: ens3
             52.96.223.2                       -- link: ens3
             52.96.228.130                     -- link: ens3
             52.96.172.98                      -- link: ens3
             52.96.111.82                      -- link: ens3
             52.96.222.194                     -- link: ens3
             52.96.91.34                       -- link: ens3

-- Information acquired via protocol DNS in 89.6ms.
-- Data is authenticated: no; Data was acquired via local or encrypted transport: yes
-- Data from: network
theuser@server:~$
      • Check your upstream DNS server can handle DNSSEC correctly
      • Delv and Dig will bring back results from the stub resolver but it should be noted these results also have the SERVFAIL flag even though they appear to bring back IP addresses successfully.
      • resolvectl
        • resolvectl seems to ignore the DNSSEC setting and still bring back errors when DNSSEC=false, this app might be using a common library that has the same bug in it. 
          theuser@server:~$ resolvectl query outlook.com
outlook.com: 52.96.222.194                     -- link: ens3
             52.96.228.130                     -- link: ens3
             52.96.172.98                      -- link: ens3
             52.96.111.82                      -- link: ens3
             52.96.229.242                     -- link: ens3
             52.96.222.226                     -- link: ens3
             52.96.91.34                       -- link: ens3
             52.96.214.50                      -- link: ens3
             52.96.223.2                       -- link: ens3

-- Information acquired via protocol DNS in 45.0ms.
-- Data is authenticated: no; Data was acquired via local or encrypted transport: yes
-- Data from: cache network

theuser@server:~$ resolvectl query protection.outlook.com
protection.outlook.com: resolve call failed: 'protection.outlook.com' does not have any RR of the requested type

theuser@server:~$ resolvectl query mail.protection.outlook.com
mail.protection.outlook.com: resolve call failed: 'mail.protection.outlook.com' does not have any RR of the requested type
    • Causes
      1. DNS lookups failing because the upstream servers are unavailable
      2. This is a known bug with systemd-resolved 253
    • Solutions
      1. Fix the DNS request to the upstream DNS server.
      2. Disable DNSSEC on systemd-resolved (on the stub resolver) until the bug is fixed and merged into the active systemd version.
        • Search on this page for Enable DNSSEC in `systemd-resolved` for information on how change this setting bbut a summary is below:
          /etc/systemd/resolved.conf

DNSSEC=true

-->

DNSSEC=false
    • Research
      • *.mail.protection.outlook.com Missing AAAA IPv6 - Microsoft Community
        • Microsoft does provide support for IPv6.
        • However, you'd need to fulfil the requirements below.
          • The source IPv6 address must have a valid reverse DNS lookup (PTR) record that allows the destination to find the domain name from the IPv6 address.
          • The sender must pass either SPF verification (defined in RFC 7208) or DKIM verification (defined in RFC 6376)
      • DNSSEC= - resolved.conf(5) — systemd-resolved — Debian bookworm — Debian Manpages
        • Takes a boolean argument or "allow-downgrade". If true all DNS lookups are DNSSEC-validated locally (excluding LLMNR and Multicast DNS). If the response to a lookup request is detected to be invalid a lookup failure is returned to applications.
        • Note that this mode requires a DNS server that supports DNSSEC.
        • If the DNS server does not properly support DNSSEC all validations will fail.
        • If set to "allow-downgrade" DNSSEC validation is attempted, but if the server does not support DNSSEC properly, DNSSEC mode is automatically disabled.

Postfix (Email / MTA)

  • Official Sites
  • SMTP Access Restrictions
  • What is Relaying?
  • Misc
  • Ports
  • SMTP Handshake, Commands and Responses
  • Postfix Server Commands
  • Diagnostics (Email Diagnostics)
    • postconf
      • See the values 
        # See actual values
postconf
postconf | grep smtpd_sasl_security_options

# See default values
postconf -d
postconf -d | grep smtpd_sasl_security_options

# Show only configuration parameters that have explicit name=value settings in main.cf. (i.e. changes)
postconf -n
postconf -n | grep smtpd_sasl_security_options
    • Testing SASL
    • SMTP Access Restrictions
      • Postfix SMTP relay and access control | postfix.org
        • Postfix has several features that aid in SMTP access rule testing:
        • soft_bounce
        • warn_if_reject (When placed before a reject-type restriction)
        • XCLIEN
      • Windows Outlook clients fails the HELO test when sending emails
        • Outlook on Windows by default only sends the computers name in the (e.g. helo=<laptop>) which can result in failures dues to policy restrictions set by your mail server. 
          May 27 07:57:04 server.example.com postfix/smtpd[1072276]: connect from router.example.com[10.0.0.1]
May 27 07:57:05 server.example.com postfix/smtpd[1072276]: NOQUEUE: reject: RCPT from router.example.com[10.0.0.1]: 504 5.5.2 <laptop>: Helo command rejected: need fully-qualified hostname; from=<testuser@example.com> to=<remoteuser@remoteserver.com> proto=ESMTP helo=<laptop>
May 27 07:57:07 server.example.com postfix/smtpd[1072276]: disconnect from router.example.com[10.0.0.1] ehlo=1 auth=1 mail=1 rcpt=0/1 quit=1 commands=4/5
        • Even if you PC has a FQDN hostname via DNS, the HELO name will still be just your computer name.
        • Solutions
          • Join a domain that has a FQDN (not tried)
          • Manually add a domain suffix (not tried)
          • Use permit_sasl_authenticated instead of relying on permit_mynetworks.
        • Postfix - `permit_networks` does not work · Issue #2174 · webmin/webmin · GitHub - This has a full technical walk through of this issue with solutions.
    • Test a SSL Certificate
  • Logs and monitoring
    • Email : cannot send emails but I can receive - Virtualmin - Virtualmin Community
      • We need to see the relevant maillog/mail.log entries, or the relevant journal entries (the postfix unit is probably the relevant one).
      • e.g. start a tail on the postfix log: 
        journalctl -fu postfix
      • And, then try to send mail. See what happens in the log. Show us what happens in the log if you don’t understand it (don’t post a million lines…we probably just need to see the two or three entries that appear right when you try to send mail).
      • the issue for references was clearly PRT/rDNS entries were wrong.
    • 18.04 - Postfix is not logging anything to journal - Ask Ubuntu
      • Maybe you should try: 
        journalctl -f -u postfix@-.service
      • If this is not working, search for more postfix-services in th 
        systemctl --all | grep post
    • Some more useful log commands 
      journalctl -t postfix/smtpd -t postfix/smtp -f

or

journalctl -u postfix -f
    • monitoring - Continuously monitor the postfix Mailqueue (real time) - Server Fault
      • I know of the commands, postqueue -p and mailq. What I am looking for is real time monitoring of the queue. Similar to when I monitor a log with tail -f.
      • You can run either of those through watch: 
        watch -n1 mailq

watch 'mailq | grep "[^A-F0-9]"'
  • SPAM
  • SPF
    It is recommended to keep Postfix as an MTA and have external softwares handle spam detection, but SPF sits inbetween as to whether Postfix should handle this because it is an email technology.
  • Postfix and TLS
  • Delivery Issues
    • General things to look at
      • You have a trusted SSL (i.e. Lets Encrypt SSL) for the domain your domain
      • DANE is setup and configured correctly, if not this will stop the remote server from talking to yours.
      • Check all the other Email antispam technologies.
      • Make sure your SMTP Access Restriction Policies are not too strict.
      • Check and monitor emails log for errors 
        journalctl -t postfix/smtpd -t postfix/smtp -f

or

journalctl -u postfix
    • 550-Verification failed for 550 Sender - Help! (Home for newbies) - Virtualmin Community
      • Check you have the following records setup correctly:
        • rDNS (Reverse DNS / PTR)
        • MX
        • SPF
        • DKIM
        • DMARC
        • DNSSEC
        • DANE
      • Most of the time your PTR record is out of your control. Your host owns the network block, not you. Some will delegate it on request, others will set it to a value of your choosing…almost never is it something that will Just Work if you set it up in your DNS server without explicit/informed cooperation from your hosting provider. = so from this just set up a ptr record at your hosting provider, if the allow that, or get them to setup a ptr record for you.
      • You should disable DNS if your using a external DNS, Suggested will then show.
      • Large Company Requirements
        • DMARC is only needed for Google and Yahoo if you are a bulk sender – Google defines that as 5000 messages per day – Yahoo only uses the term Bulk Sender.
        • If you are under 5k/day, you need EITHER SPF or DKIM, with DMARC being optional
        • if you are over 5k/day, you need all three
      • The error message “550 Verification failed for 550 Sender” typically indicates an issue with the sender’s email address or domain verification during the email sending process. This error is often encountered in email systems that utilize Sender Policy Framework (SPF) or DomainKeys Identified Mail (DKIM) authentication mechanisms to verify the authenticity of email senders.
      • Give changes you make time to propagate.
      • Sounds like masquerading/NAT? If your server is connecting to the world through a NAT router, you need the public IP to be the one that has all the necessary IP-related stuff for sending mail (PTR, SPF). You also probably need DKIM (and maybe DMARC) for strict receivers, but that’s not IP-tied, that’s a public key stored in DNS (and DMARC is a record that explains what verification methods are available from among those others and tells receivers they should reject any mail that claims to be from your domain that doesn’t meet those requirements).
    • SSL Certificate is self signed
      • This will cause you all sorts of issues becasue you need a SSL certificate from a CA.
      • You can confirm this by monitoring the log (live) to see the error shown below: 
        ### Logging command
journalctl -t postfix/smtpd -t postfix/smtp -f

### The Error
May 25 14:57:45 web.svchost.uk postfix/smtpd[867290]: connect from server.example.com[44.44.44.44]
May 25 14:57:45 web.svchost.uk postfix/smtpd[867290]: SSL_accept error from server.example.com[44.44.44.44]: -1
May 25 14:57:45 web.svchost.uk postfix/smtpd[867290]: warning: TLS library problem: error:0A000418:SSL routines::tlsv1 alert unknown ca:../ssl/record/rec_layer_s3.c:1584:SSL alert number 48:
May 25 14:57:45 web.svchost.uk postfix/smtpd[867290]: lost connection after STARTTLS from server.example.com[44.44.44.44]
May 25 14:57:45 web.svchost.uk postfix/smtpd[867290]: disconnect from server.example.com[44.44.44.44] ehlo=1 starttls=0/1 commands=1/2
      • Solution is to install the Lets Encrypt SSL Certificate for the mail (and other) services for the domain you are trying to accept mail for.
        • Virtual Server --> Manage Virtual Server --> Setup SSL Certificate --> Current Certificate --> Set as Default Services Certificate
      • “error: 14090086:SSL routines;SSL3 GET SERVER CERTIFICATE: certificate verify failed;” while HTTP transformation fails in PowerCenter - Informatica Community Support
        • Trest the certificate withe follow command: 
          openssl s_client -showcerts -connect​ host:port
 
openssl s_client -showcerts -connect 1.1.1.1:1025
      • ssl certificate - Error during openssl s_client connection, SSL alert number 48 - Server Fault
        • tlsv1 alert unknown ca = The server cannot verify the client certificate you've sent because it does not find any path to the CA's trusted by the server. 
        • These codes - the "48" - are defined in the TLS spec. E.g.section 7.2 ("Alert Protocol") in RFC 5246. 48 is "unknown_ca" which as discussed previously means it does not recognize the signer of your client certificate.
    • A successful email delivery shown in the log 
      May 25 17:06:37 server.example.com postfix/smtpd[920568]: connect from mail-dm3nam02olkn2069.outbound.protection.outlook.com[40.92.43.69]
May 25 17:06:38 server.example.com postfix/smtpd[920568]: 284463802EE: client=mail-dm3nam02olkn2069.outbound.protection.outlook.com[40.92.43.69]
May 25 17:06:43 server.example.com postfix/smtpd[920568]: disconnect from mail-dm3nam02olkn2069.outbound.protection.outlook.com[40.92.43.69] ehlo=2 starttls=1 mail=1 rcpt=1 bdat=1 quit=1 commands=7
      • You can tell it is sucessfully delivered to your server because of the line with "284463802EE: client=mail".
  • Misc Errors
    • Temporary failure in name resolution
      • Error 
        Jun 02 09:17:28 router.example.com postfix/smtpd[1720446]: warning: hostname server.example.com does not resolve to address 10.0.0.1: Temporary failure in name resolution
Jun 02 09:17:28 router.example.com postfix/smtpd[1720446]: connect from unknown[10.0.0.1]
      • Cause
        • You are using the systemd-resolved stub DNS resolver and because your router does not have it's name registered against your external IP, the DNS reolution will fail.
      • Solutions
        • Add a value into your hosts file for this mapping
          • Webmin --> Networking --> Host Addresses
          • /etc/hosts
        • Only use your router for DNS lookups and bypass systemd-resolved
        • see the systemd-resolved (DNS Resolver) for more details.
  • DNSSEC errors
    • If you do not have DNSSEC correctly setup on your server you will get these errors when using sending emails with `dane` or `dane-only`enabled. However when on `dane` emails will be delivered by standard email delivery whereas `dane-only` will fail the email.
    • `warning: DNSSEC validation may be unavailable` / `warning: received a response that is not DNSSEC validated` / DNSSEC not working
      • Error 
        May 31 10:34:05 server.example.com postfix/smtp[1530730]: warning: DNSSEC validation may be unavailable
May 31 10:34:05 server.example.com postfix/smtp[1530730]: warning: reason: dnssec_probe 'ns:.' received a response that is not DNSSEC validated
May 31 10:34:05 server.example.com postfix/smtp[1530730]: warning: TLS policy lookup for remoteserver.com/remoteserver.com: non DNSSEC destination
      • Cause
        • The Ubuntu local stub resolver is not configured to handle DNSSEC, and/or your upstream DNS server is not DNSSEC capable.
      • Solution
        • Add DNSSEC support to the DNS lookup service (i.e. `systemd-resolved`) that is being used by your system.
      • Links
        • Postfix stable release 3.5.9 and legacy releases 3.4.19, postfix-3.3.16, 3.2.21 - DNSSEC validation is needed for Postfix DANE support; this ensures that Postfix receives TLSA records with secure TLS server certificate info. When DNSSEC validation is unavailable, mail deliveries using opportunistic DANE (security level 'dane') will not be protected by server certificate info in TLSA records, and mail deliveries using mandatory DANE (security level 'dane-only') will not be made at all.
        • dnssec_probe - Postfix Configuration Parameters | postfix.org
          • The DNS query type (default: "ns") and DNS query name (default: ".") that Postfix may use to determine whether DNSSEC validation is available.
          • Possible reasons why DNSSEC validation may be unavailable:
            • The local /etc/resolv.conf file specifies a DNS resolver that does not validate DNSSEC signatures (that's $queue_directory/etc/resolv.conf when a Postfix daemon runs in a chroot jail).
            • The local system library does not pass on the "DNSSEC validated" bit to Postfix, or Postfix does not know how to ask the library to do that.
    • warning: TLS policy lookup / non DNSSEC destination / status=deferred (non DNSSEC destination)
      • Error 
        Jun 02 09:17:28 web.svchost.uk postfix/smtpd[1720446]: connect from unknown[10.0.0.1]
Jun 02 09:17:28 web.svchost.uk postfix/smtpd[1720446]: 7CE503810E0: client=unknown[10.0.0.1], sasl_method=PLAIN, sasl_username=testuser@example.com
Jun 02 09:17:28 server.example.com postfix/smtp[1720449]: warning: TLS policy lookup for remoteserver.com/remoteserver.com: non DNSSEC destination
Jun 02 09:17:28 server.example.com postfix/smtp[1720449]: 7CE503810E0: to=<remoteuser@remoteserver.com>, relay=none, delay=0.48, delays=0.38/0.01/0.1/0, dsn=4.7.5, status=deferred (non DNSSEC destination)
Jun 02 09:17:31 server.example.com postfix/smtpd[1720446]: disconnect from unknown[10.0.0.1] ehlo=1 auth=1 mail=1 rcpt=1 data=1 quit=1 commands=6
      • Cause
        • `smtp_tls_security_level` is set to `dane-only` which does not allow downgrading.
        • The target email address is in a non DNSSEC zone and cannot be delivered, so has been deferred.
      • Further information
        • Deferred emails end up in the mail queue: Webmin --> Servers --> Postfix Mail Server --> Mail Queue
        • Trying to resend or any other re-try operation will likely fail with no changes to either mail server.
      • Solution
        • Change `smtp_tls_security_level` from `dane-only` to `dane`
        • Webmin --> Servers --> Postfix Mail Server --> SMTP Client Options --> SMTP TLS client security level: dane
      • Links
  • Generic Restrictions
    • Generic restrictions that can be used in any SMTP command context, described under smtpd_client_restrictions. These are just a few examples taken from Postfix Configuration Parameters
      • permit
        • Permit the request. This restriction is useful at the end of a restriction list, to make the default policy explicit.
      • reject_unauth_pipelining
        • Reject the request when the client sends SMTP commands ahead of time where it is not allowed, or when the client sends SMTP commands ahead of time without knowing that Postfix actually supports ESMTP command pipelining. This stops mail from bulk mail software that improperly uses ESMTP command pipelining in order to speed up deliveries.
        • Postfix supports a technique known as pipelining that speeds up bulk deliveries of email by sending multiple smtp commands at once. The protocol requires that clients first check that the server supports pipelining. Many spammers send a series of commands without waiting for authorization, in order to deliver their messages as quickly as possible. reject_unauth_pipelining stops mail from bulk mail software that improperly uses pipelining in order to speed up deliveries.
      • reject
        • Reject the request. This restriction is useful at the end of a restriction list, to make the default policy explicit. The reject_code configuration parameter specifies the response code for rejected requests (default: 554).
    • Other restrictions that are valid in this context:

Cyrus SASL Authentication Server

General
  • Offcial
  • Virtualmin uses these:
  • Cyrus Config Location(s)
    • Cyrus SASL configuration file location - Postfix SASL Howto | postfix.org
      • Cyrus SASL version 2.x searches for the configuration file in /usr/lib/sasl2/.
      • Cyrus SASL version 2.1.22 and newer additionally search in /etc/sasl2/.
      • Some Postfix distributions employ a non-empty default value for cyrus_sasl_config_path to look for the Cyrus SASL configuration file in /etc/postfix/sasl/, /var/lib/sasl2/ etc. See the output of postconf cyrus_sasl_config_path and/or the distribution-specific documentation to determine the expected location.
      • Cyrus SASL searches /usr/lib/sasl2/ first. If it finds the specified configuration file there, it will not examine other locations.
What is 'SASL Authentication'?

SASL is a framework for application protocols, such as SMTP, POP3 or IMAP, to add authentication support using external packages.

  • General
    • GNU Simple Authentication and Security Layer 2.2.1
      • SASL is used by network servers (e.g., IMAP, SMTP, XMPP) to request authentication from clients, and in clients to authenticate against servers.
      • SASL is a framework for application protocols, such as SMTP or IMAP, to add authentication support. For example, SASL is used to prove to the server who you are when you access an IMAP server to read your e-mail.
      • The SASL framework does not specify the technology used to perform the authentication, that is the responsibility for each SASL mechanism. Popular SASL mechanisms include CRAM-MD5 and GSSAPI (for Kerberos V5).
      • Typically a SASL negotiation works as follows. First the client requests authentication (possibly implicitly by connecting to the server). The server responds with a list of supported mechanisms. The client chose one of the mechanisms. The client and server then exchange data, one round-trip at a time, until authentication either succeeds or fails. After that, the client and server knows more about who is on the other end of the channel.
      • For example, in SMTP communication happens like this: 
        250-mail.example.com Hello pc.example.org [192.168.1.42], pleased to meet you
250-AUTH DIGEST-MD5 CRAM-MD5 LOGIN PLAIN
250 HELP
AUTH CRAM-MD5
334 PDk5MDgwNDEzMDUwNTUyMTE1NDQ5LjBAbG9jYWxob3N0Pg==
amFzIDBkZDRkODZkMDVjNjI4ODRkYzc3OTcwODE4ZGI5MGY3
235 2.0.0 OK Authenticated
        • Here the first three lines are sent by the server and contains the list of supported mechanisms (DIGEST-MD5, CRAM-MD5, etc). The next line is sent by the client to select the CRAM-MD5 mechanism. The server replies with a challenge, which is a message that can be generated by calling GNU SASL functions. The client replies with a response, which also is a message that can be generated by GNU SASL functions. Depending on the mechanism, there can be more than one round trip, so do not assume all authentication exchanges consists of one message from the server and one from the client. The server accepts the authentication. At that point it knows it is talking to a authenticated client, and the application protocol can continue.
      • Essentially, your application is responsible for implementing the framing protocol (e.g., SMTP or XMPP) according to the particular specifications. Your application uses GNU SASL to generate the authentication messages.
    • Simple Authentication and Security Layer - Wikipedia - Simple Authentication and Security Layer (SASL) is a framework for authentication and data security in Internet protocols. It decouples authentication mechanisms from application protocols, in theory allowing any authentication mechanism supported by SASL to be used in any application protocol that uses SASL.
    • Thunderbird:Supported authentication methods - MozillaWiki
    • Security & Authentication: SSL vs SASL - Stack Overflow
    • SASL, What is it? Why do I need it? - Virtualmin - Virtualmin Community
      • SMTP has had several authentication mechanisms available over the years, the current one is called Simple Authentication and Security Layer.
      • The SMTP server (Postfix, in most cases) uses SASL (specifically, Cyrus saslauthd).
    • Introduction to Simple Authentication Security Layer (SASL) - Developer's Guide to Oracle Solaris 11 Security - SASL provides developers of applications and shared libraries with mechanisms for authentication, data integrity-checking, and encryption. SASL enables the developer to code to a generic API. This approach avoids dependencies on specific mechanisms. SASL is particularly appropriate for applications that use the IMAP, SMTP, ACAP, and LDAP protocols, as these protocols all support SASL. SASL is described in RFC 2222.
    • encryption - Security & Authentication: SSL vs SASL - Stack Overflow - SASL is essentially an indirection layer to allow for pluggable authentication systems and data security in existing application protocols (e.g LDAP, SMTP, Subversion, ...), although these protocols need to be aware of this extension (e.g. SMTP auth). Whether and how it provides secure authentication and data encryption depend heavily on what underlying mechanism is used within this framework. Here is an example from the svnserve documentation: "The built-in CRAM-MD5 mechanism doesn't support encryption, but DIGEST-MD5 does".
  • Cyrus
    • Cyrus SASL — Cyrus SASL 2.1.28 documentation - Simple Authentication and Security Layer (SASL) is a specification that describes how authentication mechanisms can be plugged into an application protocol on the wire. Cyrus SASL is an implementation of SASL that makes it easy for application developers to integrate authentication mechanisms into their application in a generic way.
    • What SASL is - System Administrators — Cyrus SASL 2.1.28 documentation
      • SASL, the Simple Authentication and Security Layer, is a generic mechanism for protocols to accomplish authentication. Since protocols (such as SMTP or IMAP) use SASL, it is a natural place for code sharing between applications. Some notable applications that use the Cyrus SASL library include Sendmail, Cyrus imapd, and OpenLDAP.
      • Applications use the SASL library to tell them how to accomplish the SASL protocol exchange, and what the results were.
      • SASL is only a framework: specific SASL mechanisms govern the exact protocol exchange. If there are n protocols and m different ways of authenticating, SASL attempts to make it so only n plus m different specifications need be written instead of n times m different specifications. With the Cyrus SASL library, the mechanisms need only be written once, and they’ll work with all servers that use it.
  • Postfix
    • Postfix SASL Howto | postfix.org
      • Clean explanation what this is and what it is for.
      • Currently the Postfix SMTP server supports the Cyrus SASL and Dovecot SASL implementations.
      • The Cyrus SASL framework supports a wide variety of applications (POP, IMAP, SMTP, etc.). Different applications may require different configurations. As a consequence each application may have its own configuration file.
    • sasl - Can postfix enforce reject_unknown_helo_hostname before permit_sasl_authenticated? - Server Fault
    • Postfix And SASL - Debian Wiki - Simple Authentication and Security Layer (SASL) with Postfix SMTP. Intructions for both Cyrus SASL and Dovecot SASL. 
      pwcheck_method: saslauthd
mech_list: CRAM-MD5 DIGEST-MD5 LOGIN PLAIN
    • Postfix with SASL - ArchWiki
      • SMTP protocol specifications include a possibility for user authentication, but do not provide the exact details of protocol message exchange, deferring instead to the SASL (Simple Authentication and Security Layer) standard (see RFC 4954 and RFC 4422).
      • SASL is a generic authentication framework for authentication mechanisms, of which there are many, and each of them has its own particular procedure that prescribes the necessary cryptographic steps to perform with the authentication data and messages to exchange over the connection.
      • Therefore, in order to avoid imposing artificial limits on what authentication mechanisms can be used with it, Postfix, by itself, does not authenticate SMTP users with usernames and passwords, or via any other means. It offloads this task to a SASL implementation, which has to be installed separately.
      • SASL authentication daemon is responsible both for the policy (i.e. where valid usernames and secrets such as passwords are kept) and mechanism (how exactly clients supply credentials).
    • How to enable user authentication for a Postfix SMTP server with SASL | xmodulo.com
      • This tutorial will focus on setting up a Postfix SMTP server to use Dovecot SASL for user authentication. As Dovecot provides mechanisms for user authentication, Postfix will simply ask Dovecot to do the work for it. That way, there is no need to re-invent the wheel.
  • RFC
  • Dovecot SASL for Postfix
  • Dovecot SASL
    • SASL — Dovecot documentation
      • SASL stands for “Simple Authentication and Security Layer”. SASL itself is nothing more than a list of requirements for Authentication (SASL) Mechanisms and protocols to be SASL-compatible as described in RFC 4422. IMAP, POP3, SMTP, and ManageSieve protocols all have support for SASL.
      • Many people confuse SASL with one specific SASL implementation: the Cyrus SASL library.
      • Dovecot has its own SASL implementation which could (one day) be separated from Dovecot itself to “compete” against Cyrus SASL library as an alternative implementation.
      • Dovecot can be used as the SASL server for several external SMTP/Submission servers. See SMTP AUTH.
    • Authentication (SASL) Mechanisms — Dovecot documentation
      • The simplest authentication mechanism is PLAIN. The client simply sends the password unencrypted to Dovecot. All clients support the PLAIN mechanism, but obviously there’s the problem that anyone listening on the network can steal the password. For that reason (and some others) other mechanisms were implemented.
      • Today however many people use SSL, and there’s no problem with sending unencrypted password inside SSL secured connections. So if you’re using SSL, you probably don’t need to bother worrying about anything else than the PLAIN mechanism.
      • Another plaintext mechanism is LOGIN. It’s typically used only by SMTP servers to let Outlook clients perform SMTP authentication. Note that LOGIN mechanism is not the same as IMAP’s LOGIN command. The LOGIN command is internally handled using PLAIN mechanism.
      • By default only PLAIN mechanism is enabled. To use more, edit your /etc/dovecot/conf.d/10-auth.conf and set: 
        auth_mechanisms = plain login cram-md5
    • auth_mechanisms - Dovecot Core Settings — Dovecot documentation
      • Default: Plain
      • Values: plain, login, digest-md5, cram-md5, ntlm, rpa, apop, anonymous, gssapi, otp, skey, gss-spnego
      • Here you can supply a space-separated list of the authentication mechanisms you wish to use.
    • Authentication mechanisms vs. password schemes - Authentication — Dovecot documentation - Authentication mechanisms and password schemes are often confused, because they have somewhat similar values. For example there is a PLAIN auth mechanism and PLAIN password scheme. But they mean completely different things.
    • Password Schemes — Dovecot documentation - Password scheme means the format in which the password is stored in Password databases (passdb). The main reason for choosing a scheme other than PLAIN is to prevent someone with access to the password database (such as a hacker) from stealing users’ passwords and using them to access other services.
    • HowTos/postfix_sasl | wiki.centos.org
      • By default, postfix uses the $mynetworks parameter to control access, i.e. who can send or relay mail through the mail server. There is no other authentication performed other than checking that the IP address of the user trying to send mail is part of a trusted network as specified in $mynetworks.
      • If you are only implementing a mail server where all your users are based on the same network then it is unlikely that you will need to use SASL or SSL/TLS. However, if there are mobile users that wish to use the mail server whilst away from base, we need a mechanism to authenticate them as trusted users so that they are able to send mail through the mail server.
      • SASL (Simple Authentication and Security Layer) provides a mechanism of authenticating users using their username and password. Probably the most well known implementation of SASL is provided by the Cyrus SASL library, but dovecot also has it’s own SASL implementation built in, and as we are already running dovecot we may as well use it for SASL rather than having to install and configure another package.
    • Does dovecot use Cyrus or it's own SASL on my Virtuamin installation - Virtualmin - Virtualmin Community
      • You can’t us CRAM-MD5 (or any of those other options) with system users without also storing plaintext passwords. The way the email protocols treat passwords and the way Linux treats passwords don’t have any overlap, so there’s gotta be a plaintext password somewhere…Virtualmin sets up SSL on all mail protocols (and we recommend you use them) so passwords are not transmitted in plain text. We don’t support any of those other options.
      • From the Dovecot docs:
        • "Non-plaintext mechanisms have been designed to be safe to use even without SSL encryption. Because of how they have been designed, they require access to the plaintext password or their own special hashed version of it. This means that it’s impossible to use non-plaintext mechanisms with commonly used DES or MD5 password hashes.
      • If you want to use more than one non-plaintext mechanism, the passwords must be stored as plaintext so that Dovecot is able to generate the required special hashes for all the different mechanisms. If you want to use only one non-plaintext mechanism, you can store the passwords using the mechanism’s own Password Schemes."
      • We plan a refactor of the mail stack, maybe for Virtualmin 8 (development starting later this year), which likely ends this particular dichotomy by severing “mail” and “system” users. That has far-reaching implications, but is probably better for most use cases; easier to scale across multiple systems, for instance. That may wait until JMAP is more mature, since that will also require a mail stack refactor (which would probably involve dropping Dovecot in favor of Cyrus).

Postgrey (Email /Greylisting)

Postgrey greylisting implementation for Postfix.

  • Official Sites
  • General
  • What is greylisting
    • What is greylisting and how does it work? | Scrubby.io - Explore what greylisting is and how it protects your email from unwanted spam. Explore the workings, advantages, and challenges of this effective email security tool.
    • Understanding Greylisting: An Effective Email Spam Filter - Explore how greylisting helps fight spam and ensures only legitimate emails make it to your inbox. Understand this powerful business tool with Captain Verify.
    • Greylisting (email) - Wikipedia
    • 50 shades of spam: Ultimate guide to email greylisting - MailerCheck - Email greylisting is an effective way of stopping spammers while allowing legit senders to deliver their messages. Learn what it is and how to avoid it.
    • Greylisting: The Next Step in the Spam Control War - Greylisting is a new method of blocking significant amounts of spam at the mailserver level, but without resorting to heavyweight statistical analysis or other heuristical (and error-prone) approaches. Consequently, implementations are fairly lightweight, and may even decrease network traffic and processor load on your mailserver.
    • PostfixGreylisting - Community Help Wiki | Ubuntu
      • Greylisting is a spam reduction technique that can be very effective. It works by temporarily rejecting from client machines that are unknown to the server's greylisting service. 
      • If the client is standards-compliant, it will attempt to re-send its message after its initial failed smtp session, and your receiving mail server will accept it. The client is then added to a list of known clients, and will not be delayed in the future. This means that the first e-mail from an unknown client will be delayed, but subsequent ones will be processed right away.
      • Most spam mailers, on the other hand, do not re-send messages after failed smtp sessions. Thus, in theory, greylisting effectively blocks the majority of spammers.
    • Greylisting HOW TO - HowTos/postgrey | CentOS Wiki
      • The principle of greylisting works on the basis that much spam is sent by spambots and other non RFC compliant MTAs.
      • Nice diagram.
  • Virtualmin
  • milter-greylist
  • Config File locations
    • Original
      • /usr/share/postgrey/postgrey-default
      • /usr/share/postgrey/whitelist_clients
      • /usr/share/postgrey/whitelist_recipients
    • Virtualmin
      • /etc/default/postgrey
      • These are backed up in the `Virtualmin Settings` backup when you select `Mail server settings`
        • /etc/postgrey/whitelist_clients
        • /etc/postgrey/whitelist_recipients

Dovecot (IMAP/POP3)

Dovecot is an open source IMAP and POP3 email server for Linux/UNIX-like systems, written with security primarily in mind. Dovecot is an excellent choice for both small and large installations. It’s fast, simple to set up, requires no special administration and it uses very little memory.

  • Official
  • Diagnostics
    • See the current config values from /etc/dovecot/dovecot.conf (and possibly others)
      doveconf

doveconf | grep cyrus_sasl_config_path
    • See the default values
      doveconf -d

doveconf -d | grep cyrus_sasl_config_path
    • Missing IMAP Folders after cPanel Account Migration - Help! (Home for newbies) - Virtualmin Community
      • Q: After moving accounts from cPanel to Virtualmin, I’ve finding some e-mail related things did not migrate (e-mail filters, for example) while others seem to have migrated partly. The most worrisome is IMAP folders. Some accounts came over cleanly with mailboxes’ full set of folders, whereas others have those folders within Usermin, but some or all of them do not show up in macOS or iOS mail clients.
      • A: doveadm-force-resync | Dovecot CE - Under certain circumstances it may happen, that dovecot(1) is unable to automatically solve problems with mailboxes. In such situations the force-resync command may be helpful. It tries to fix all problems. For sdbox and mdbox mailboxes the storage files will be also checked.

Procmail (Mail Filter)

Procmail is like a conveyor belt on which emails are placed at one end by Postfix, as the email goes along the belt various entities such as SpamAssassin and ClamAV act upon them at the various levels (Global, Domain, Mailbox) changing their headers if issues are found, they do not dleete or change the destination of the emails. It is Procmail rules that decides where the email is delivered, is forwarded to or deleted at anypoint in the conveyor belt.

  • General
    • Procmail Mail Filter | Webmin
      • This page explains how to use the Procmail program and Webmin to filter and deliver email coming into your system.
      • Procmail is a powerful program for filtering and re-directing email that would normally be sent to users’ mailboxes. It can be used at both the system level to filter message for all users on your system, on a per-user basis, or both.
      • Unlike normal Sendmail aliases, Procmail can be used to deliver messages differently depending on their headers and content.
    • procmail - Wikipedia - procmail is an email server software component — specifically, a message delivery agent (MDA). It was one of the earliest mail filter programs. It is typically used in Unix-like mail systems, using the mbox and Maildir storage formats.
    • GitHub - BuGlessRB/procmail - The mail sorting program.
    • Procmail is an MDA (Mail Delivery Agent), not an MTA (Mail Transport Agent)
    • procmail - mail delivery agent - LinuxLinks - Procmail is a mail delivery agent (MDA) or mail filter, a program to process incoming emails on a computer. Typically invoked from a mail transfer agent.
  • Config file
    • There is a single config file which holds all of the Procmail filter rules. 
      /etc/procmailrc
    • The rules are processed in order.
    • Webmail --> Servers --> Procmail Mail Filter

SpamAssassin (Mail Filter)

Filtering Mode

In Virtualmin you can configure SpamAssassin Mail Filtering in one of 2 modes which have the advantages and disadvatages as outlined below:

  • spamassassin (Standalone program)
    • Uses global configuration and default rule set.
    • Allows for per-domain settings
  • spamc (Client for SpamAssassin filter server spamd)
    • Uses global configuration and default rule set.
    • No per-domain settings
    • If your system is going to host domains that will receive a large amount of email, filtering incoming messages for spam can generate significant CPU load. This is due to CPU use by the SpamAssassin mail filter when it is started, which can be avoided by running the SpamAssassin filter server spamd in the background. This consumes RAM, but reduces CPU load and makes mail processing faster.

This is mode is selected during the `POST-Setup Wizard` but can be changed afterwards using:

  • Swap between Global and Per-domain
    • Virtualmin --> Email Settings --> Spam and Virus Scanning --> SpamAssassin client program
    • This will cycle through all virtual servers and make the relevant changes.
Heirarchy

The config files are processed in the order below and each section is only processed if the current Virtualmin settings allow.

  1. SpamAssassin (Default rules) 
    /usr/share/spamassassin
    • Default Rule set
    • This is replaced during upgrades
    • Do not use this to add rules
  2. Webmin (Global) 
    /etc/spamassassin/local.cf
    • Webmin --> Servers --> SpamAssassin Mail Filter
    • Add rules here you want applied globally.
    • /etc/mail/spamassassin/ is a symbolic link to /etc/spamassassin/, therefore /etc/mail/spamassassin/local.cf is symbolically linked to (sort of) /etc/spamassassin/local.cf
    • Procmail Spam Delivery
      • After these SPAM rules have been processed, these Procmail delivery rules are applied.
      • Webmin --> Servers --> SpamAssassin Mail Filter --> Procmail Spam Delivery
      • /etc/procmailrc
  3. Virtual Server (Per-domain)
     
    /etc/webmin/virtual-server/spam/[vserver_id]/virtualmin.cf
    • Virtualmin --> Mail Options --> SpamAssassin Configuration
    • Add rules that are specific to this domain
    • This is not a copy of /etc/spamassassin/local.cf or a symbolic link to it
    • spamassassin mode needs to be set to "Standalone program"
    • Procmail Spam Delivery
      • After these SPAM rules have been processed, these Procmail delivery rules are applied.
      • Virtualmin --> Mail Options --> SpamAssassin Configuration --> Procmail Spam Delivery normally
      • /etc/webmin/virtual-server/procmail/[vserver_id]
  4. Usermin (Mailbox Users) 
    # Virtual Server Owner
/home/[vserver_owner]/.spamassassin/user_prefs

or

# Mailbox User
/home/[vserver_owner]/homes/[mailbox_user]/.spamassassin/user_prefs
    • Usermin --> Mail --> SpamAssassin Mail Filter
    • spamassassin mode needs to be set to "Standalone program"
    • By default this is off for security reasons, in particular for spamc/spamd because spamd runs as root.
    • For this feature to be enabled a couple of things need to be enabled
      1. Allow mailbox users to create mail filters
      2. Allow users to define tests
        • This can be setup in a couple of places depending on whether or not you want to set it globally.
          • Webmin --> Servers --> SpamAssassin Mail Filter --> Header and Body Tests --> Swich to advanced mode --> Allow users to define tests?
            • This adds "allow_user_rules 1" to the servers global file /etc/spamassassin/local.cf which will apply to all virtual servers unless overridden by their individual configs.
            • This will apply globally.
          • Virtualmin --> Mail Options --> SpamAssassin Configuration --> Header and Body Tests --> Swich to advanced mode --> Allow users to define tests?
            • This adds "allow_user_rules 1" to the currently selected virtual server's relevant file /etc/webmin/virtual-server/spam/[vserver_id]/virtualmin.cf
            • This will apply per-domain
        • "Header and Body Tests" icon is missing in Usermin
          • Allow users to define tests? (allow_user_rules) needs to be enabled globally or for the relevant domain.
    • Procmail Spam Delivery
      • After these SPAM rules have been processed, these Procmail delivery rules are applied.
      • Usermin --> Mail Options --> SpamAssassin Mail Filter --> Procmail Spam Delivery normally
        • This is curently missing on my Usermin. This is probably a permission issue.
      • /home/[vserver_owner]/homes/[mailbox_user]/.procmailrc

 Notes

  • Config Files Missing
    • Depending of the SPAM fitering mode you have selected for Virtualmin and whether mailbox users can make their own SPAM filtering rules, not all config files will exist. usr/share/spamassassin and /etc/spamassassin/local.cf will always exists and be used.
  • SpamAssassin test scores
    • SPF_FAIL on the domain overwrites the SPF_FAIL from the global local.cf, it does not add additional points.
    • This implies for config files, the last one wins.
  • Tests
    • These are applied in order.
    • The points are added to the SPAM score.
General
  • Official
  • General
    • Spam and Virus Scanning | Virtualmin — Open Source Web Hosting Control Panel - Virtualmin allows you to enable spam and virus scanning for email on a per-virtual-server basis, and to configure what happens to email classifies as spam or virus-laden. Under the hood, it scans email using the popular SpamAssassin package for spam detection, and ClamAV for viruses.
    • SpamAssassin Mail Filter | Webmin
      • SpamAssassin Mail Filter is a powerful program for detecting un-wanted spam messages based on their headers and content. It uses a complex set of built-in rules to determine if an email is spam or not, and can also consult other databases of known spam message texts and mail servers used for sending spam.
      • However, the spamassassin program itself does not perform any real filtering, instead it just takes email as input, adds special headers indicating if the message is spam or not and then writes it out again.
    • Is forwarded mail filter by spamassassin first? - Virtualmin - Virtualmin Community
      • Q:
        • I have a couple accounts that forward to external addresses and I am wondering if email sent to those accounts gets scanned for spam by spamassassin before being forwarded? This is important because if not in the eyes for the external servers all that spam now appears to originate(in some respects) from my server. If it doesn’t scan is there a way to address this? Thanks
      • A:
        • Depending on how the forwarding is setup – it may or may not get scanned first. If you set it up through Usermin (on port 20000), it should be scanned first in that case.
        • If you want it to run emails through SpamAssassin and ClamAV, you’d want to setup forwarding using Usermin.
  • Settings
    • Virtualmin --> Email Settings --> Spam and Virus Scanning
      • The spam and virus scanning options set on this page will be applied to all virtual servers with filtering enabled. Any per-server settings will be overridden.
      • Maximum message size to process: unlimited
        • Updates:
          • /etc/webmin/virtual-server/procmail/[vserver_id] (all of them)
          • /etc/webmin/virtual-server/config
    • Filtering Defaults
  • Filtering Mode (standalone / spamc)
    • Spam Prevention - Per user or just one global process (spamc) - Virtualmin - Virtualmin Community
      • popmay
        • /etc/mail/spamassassin/local.cf is the place of choice for site-wide application of a rule. Rules placed here get applied no matter what user invokes SpamAssassin.
        • ~/.spamassassin/user_prefs is best if you want to have a rule only run when a particular user runs SA.
        • Note: if you use spamd, rules placed in user_prefs will be IGNORED by default. If you add the allow_user_rules option to your local.cf you can get spamd to honor them. However, before you enable it, you should know that this is disabled by default for security reasons. In theory a malicious local user might be able to exploit spamd with a clever regex and gain root permissions. I know of no specific vulnerabilities of this type in spamassassin at this time, but it is a possibility. I’d only turn this on if you trust your local users not to try to hack root.
      • Shirehosting
        • Spamd is a global daemon it runs once with the same set of rules for the whole server. spamc can be set up to run per user rules. as far as I remember it is one or the other not both.
        • Spamd is faster and possibly uses lower resources as its always loaded. spamc however can be an issue as if 100 email arrive at the same time you will spin up 100 spamassassin threads at x meg of ram each. This take time as its reloaded from config each time. This put a huge load on CPU and possibly ram. If you only receive 1 or 2 email at the same time use what you like but you could very easily run out of ram and cpu if 100 or more email show up together and possibly crash the server.
      • popmay
        • Spamc is the client half of the spamc/spamd pair. Spamed runs as root. Allowing a malicious user to put code in a root app config could really have bad results!
        • This is why Webmin/Virtualmin will set up user_prefs with spamassassin (Standalone program) not spamc.
    • Spam and Virus Scanning | Virtualmin — Open Source Web Hosting Control Panel
      • Virtualmin allows you to enable spam and virus scanning for email on a per-virtual-server basis, and to configure what happens to email classifies as spam or virus-laden.
      • Internally, Virtualmin creates an /etc/procmailrc file that in turn runs a Procmail include file under /etc/webmin/virtual-server/procmail directory, depending on the domain to which each email received is sent. This then invokes the spamassassin and clamscan commands, then uses their output to decide if email should be delivered to a special folder or deleted.
      • SpamAsssassin is run with command-line parameters that tell it to use configuration files under /etc/webmin/virtual-server/spam, which can be different for each domain. This way, domain owners can customize their own SpamAssassin rules, spam levels and message modification settings.
    • Spam and Anti-Virus Scanning – Virtualmin
      • Virtualmin allows you to enable spam and virus scan emails on a per-virtual-server basis, and to configure what happens to email classifies as spam or virus-laden.
    • spamassassin level per user - Virtualmin - Virtualmin Community
      • Are you asking how to make SpamAssassin work on a per-user basis?
        • Virtualmin --> Email Messages --> Spam and Virus Scanning --> SpamAssassin client program: spamassassin (Standalone program)
        • Once you do that, SpamAssassin will check for a config file in $HOME/.spamassassin/user_prefs each time it delivers an email.
      • it is possible to edit SpamAssassin settings on a per-user basis for anyone with a Virtualmin login
        • Virtualmin --> Mail Options --> SpamAssassin Configuration
        • This also assumes `spamassassin (Standalone program)` is selected
      • Jamie Said:
        • When a virtual server owner edits his spamassassin config, it actually updates files in /etc/webmin/virtual-server/spam . These get used as the global config when spamassassin is run as the user who is receiving email, and are combined with the user's personal config in ~/.spamassassin
        • The /etc/webmin/virtual-server/spam/$DOMAINID directory is initially populated with a copy of the global config, which the domain owner can then override.
    • Spamassassin - Create mail filters per mailbox - Virtualmin - Virtualmin Community
      • According to the user manual, the SpamAssassin client can be set as spamassassin (standalone) or spamc (client for SA filter server spamd). When the last one is used, it only reads the global configuration.
      • Virtualmin offers the possibility to use preferences per virtual server. In this case spamassassin must be set as the client. There is an Allow mailbox users to create mail filters option in the Virtualmin > Email Settings > Spam and Virtus Scanning section.
      • How does this option work at the mailbox level? Which files are responsible after it is enabled. If this is the only responsible file /home/domain/.spamassassin/user_prefs, it applies to all mailboxes not at the individual mailbox level.
      • If I go to Virtualmin > Services > SpamAssassin Configuration and I would like to edit the configuration file it opens this file /etc/webmin/virtual-server/spam/[numbers]/virtualmin.cf. What is the scope of the user_prefs file in this case?
    • How to add a spamassassin rule to block all mails that contain a certain word - Webmin - Virtualmin Community
      • Eric
        • I use spamc myself, as it just uses one running SpamAssassin service instance. Then, for each incoming email, a small spamc process is launched to communicate with it.
        • The other option launches a full SpamAssassin process for each incoming email, but has no resident SpamAssassin service.
        • The second option I think is best on servers with extremely low email traffic, which are also low on RAM. However, the second option also allows per-domain SpamAssassin settings, rather than global settings.
    • Virtualmin Professional - Email Filtering - ClamAV & SpamAssassin - Virtualmin - Virtualmin Community
      • Yes. Jamie wrote a very clever bit of code to implement virus and spam filtering on a per-domain basis. Both SpamAssassin and ClamAV are installed during the installation process and are pre-configured for use by Virtualmin. As stated in the FAQ, this should Just Work.
  • Using a Database
  • DNSBLs (aka. RBL)
    • DnsBlocklists - SPAMASSASSIN - Apache Software Foundation
      • DNS Blocklists are a common form of network-accessible database used in spam detection. They're also referred to as "DNSBLs", "DNS Blacklists" and "RBLs". (The latter usage is incorrect; see RBL.)
      • Q: My queries to a DNS-blocklist were blocked. What does this mean?
      • A: DNS-Blocklists often run on the "free for some" model and/or they may limit the number of queries you can perform to maximize resources.
      • Q: This documentation doesn't seem to cover how to configure DNS-Blocklists. It says "Support for these is built-in" but I can't believe that all free BL's is called each time a mail is beeing checked. There must be a way to configure which to use.
      • A: You're right. You might look at the Mail::SpamAssassin::Conf documentation page which I admit doesn't really say how to configure which DNSBL to use, or the rules file 20_dnsbl_tests.cf, for internal details, but no clear examples of how to configure the inclusion of various DNSBLs either. For the latest list of DNSBLs you want to be using a recent SpamAssassin version (3.4.1 at the time of this correction) and sa-update, for the same reason that you wouldn't use an out-of-date virus scanner, but that also doesn't really have anything to do with the question.
      • Webmin  --> Servers --> SpamAssassin Mail Filter --> Edit Config Files --> /etc/spamassassin/local.cf
        • If you don't want any DNSBLs used, put a line like in your local.cf 
          skip_rbl_checks 1
        • To eliminate the use of a particular DNSBL, set the score to zero. Put lines like 
          score RCVD_IN_RFCI 0 score RCVD_IN_ORBS 0 score RCVD_IN_DSBL 0
    • FAQs | How you can use the free Spamhaus Blocklists - Find a definition and frequently asked questions for postmasters and troubleshooting of Domain Name System Blocklists (DNSBL).
    • SpamAssassin - How to use dnswl.org in your spam filter – dnswl.org
    • I keep getting spam - #38 by DarkCorner - Virtualmin - Virtualmin Community
    • Enable and Test URIDNSBL DNS Blocklists with SpamAssassin - SpamAssassin, the most popular open-source spam fighting software, is used by email server administrators to reduce spam and improve user productivity. One of the features of SpamAssassin is dynamic lookups of domain names to see if they are on a DNS blocklist maintained by web authorities.
  • Filters, Rules and Scores
  • Training
    • Enable smaptrap@ and hamtrap@ emails
      • Virtualmin --> Mail Options --> Spam and Virus Delivery --> Create spamtrap and hamtrap email aliases
    • Spamassassin Bayes DB - using SA-Learn, autolearn - Webmin - Virtualmin Community
      • I want to allow the users to train the spamassassin by themselfs. For this I created a Imap Folder, which gets scanned by SA-Learn by a cronjob. 
        sa-learn -u user@server.tld --spam /home/server/homes/user/Maildir/.spam/{cur,new} --progress
      • which learns the spam into the bayes db. The Spam is recognized correctly if i do a manual scan with: 
        spamassassin -D -p /home/server/homes/user/ -e < "$email" > /dev/null 2>&1
      • But still spam messages are reaching my inbox, which are scanned against a bayes db according to the headers, but it seems that spamassassin is not using the users DB.
      • Spamassassin is configured as “standalone” in the moment.
    • Customising Spam Assassin - more aggressive filtering recommendations - Virtualmin - Virtualmin Community
      • Includes pointers for better rules and how to improve the learning aspect using cron jobs.
    • How to run spamassassin and move to spam folder on a regular basis, not just when email comes in? - Virtualmin - Virtualmin Community
      • Q:
        • I have quote spam issues that spamassassin does not pick up.
        • One such reason could be that when it arrives in my inbox, its not on blacklists yet, so the score is below 5. If I wait a short time, then run spamassassin from the command line, and check blacklists, its on a spam blacklist then and the score is much higher than 5. But by then its in my inbox.
        • I dont check my inbox that often, specially overnight, so I want is to run spamassassin on as inbox at regular intervals, like every 10 mins, and if its scored as spam, move it to the spam folder. That way, it will mark as spam all the messages put on a blacklist shortly after I receive them, and this will mean I wont see many of these spam messages seen as I would only check my inbox a few times a day.
      • A:
        • Use “spamtrap” and/or “Mark As Spam” (inside Usermin). These two options which essentially do the same thing send the message to SpamAssassin on demand to be scored and therefore improve overall detection in the future.
        • Remember, SpamAssassin needs a lot of “spam” and “ham” data to accurately predict spam / ham in the future so don’t expect anything overnight.
    • Spamassassins Bayesian learning filter - Virtualmin - Virtualmin Community
      • Q:
        • I was wondering how the learning filter for spam works. We had some reports from differen domains of receiving a lot of spam. We made some adjustments to the spamfilter which made the spam mails become way less. However we noticed that the learning filter only works for the specific domain. Instead we would like the learning filter to work for all domains on the server. It is a lot of work to go through all domains by hand to make sure the learning filter works the way we want to. If the learning filter works for all domains you only need to configure it for one domain and the rest can use it as well. Otherwise you still kind of have the problem that specific kinds of spam will be received by the other domains on the server.
      • A:
        • It depends on how you have SpamAssassin configured. If you’ve configured it to allow “per-domain” settings, then each domain will have to establish it’s patterns.
        • If you’ve configured it server wide, then all reports will affect all domains respectively.
        • Keep in mind, it takes time for SpamAssassin to learn about spam in order to prevent as much false positives.
        • Make sure everyone is reporting spam regularly.
    • How to train SpamAssassin? - Help! (Home for newbies) - Virtualmin Community
      • There are a couple things you can do to improve spam detection.
        • If you use Usermin, click “Delete Spam” which should tell Spam Assassin that this message is spam and help train it to find future spam.
        • If you use an email client like Thunderbird or Outlook, you can forward spam messages to “spamtrap@yourdomain.com”. This sends the message a special alias managed by Spam Assassin which trains it.
    • Hints for getting spamtrap/hamtrap to function correctly (spamassassin) - Help! (Home for newbies) - Virtualmin Community - I had significant trouble getting hamtrap/spamtrap to function in my environment. I’ve solved the various issues and I am posting this in hopes that it will help others (and possibly lead to the necessary bug fixes and/or official documentation changes. I’m happy to help as I’m able.)
    • How to train spamassassin using WM/VM ? - Help! (Home for newbies) - Virtualmin Community
      • I can’t seem to find the path where messages are stored so I can use salearn to teach spamassassin what is and is not spam. What is the default path? I am using the default setup for WB/VM?
      • This gives you the location and some commands to traing HAM and SPAM.
        /home/domain/homes/user/Maildir

sa-learn --no-sync --ham /home/domain/homes/user/Maildir/.INBOX.ham/{cur,new}

sa-learn --no-sync --spam /home/domain/homes/user/Maildir/.INBOX.spam/{cur,new}
    • How To Train SpamAssassin | faisal.com - This is an overview of how to train SpamAssassin to more effectively catch spam.
    • Spamassassin and Virtualmin help [Solved] - Virtualmin - Virtualmin Community
      • Q:
        • I have Ubuntu Server 14.04 with a Virtualmin install and several virtual servers. One of these servers has a single mail user, and that user gets a huge amount of spam. I have created Ham and Spam folders on that mail account, and I’m trying to set up a cron job that runs sa-learn twice a day on them to continually train Spamassassin. 
          sa-learn --spam /home/<server>/homes/<user>/Maildir/.Spam/cur sa-learn --ham /home/<server>/homes/<user>/Maildir/.Ham/cur
        • I know training spamassassin on a systemwide basis is a bad idea. The problem is that I’m not sure how to make these commands applicable either to only the virtual server or to only the particular user (either solution would be acceptable). Should I run the command as sudoed to that user? Do I use the -u flag in sa-learn? In either case, do I specify user.server or just server? Or am I missing the correct method completely?
      • A:
        • I think I figured this one out myself by trial and error, and I’ll post it in case anyone has the same question. I don’t know if this is the best solution, but the solution that works for me is by formatting the commands like this and putting them into the root’s crontab: 
          sa-learn -u . --spam /home//homes//Maildir/.Spam/cur
sa-learn -u . --ham /home//homes//Maildir/.Ham/cur
    • spamassassin learning from spam folder [#56214] | Virtualmin
      • Q: I want to optimize spamassassin and know that this is possible to run a sa-learn command with the option --spam through the spam folder. I know that spam mails are put there from spamassassin, but spamassassin does not recognize the spam mails which the user - which uses IMAP - has moved there. Also the other way round this command with the --ham option should be used in the inbox, if the user moved some good mails from the spam folder to the inbox.
      • A: One command you might find useful is this which will dump all his spam as output. 
        virtualmin list-mailbox --domain whatever.com --user bob --folder Spam
  • Training (cPanel) (from old host help thread)
    • Instructions
      • When our clients are receiving too much spam, we recommend they train SpamAssassin to better identify the type of spam they are receiving.
      • This is done by creating 2 folders using IMAP or webmail, in any email account that falls under the cPanel account that is receiving the excess spam.
      • The 2 folders should be named ".HAM-TRAIN" and ".SPAM-TRAIN", where each of the folders should be populated with at least 200 messages.
      • In the .HAM-TRAIN folder, you should place the legitimate messages received and place the spam messages in the .SPAM-TRAIN folder.
      • Once both folders are populated, let us know so we can perform the training which affects the entire cPanel acount, which means this training and folder creation is not necessary to redo on a per email or domain basis.
    • Question 1
      • above we talk about the 2 folders for spam training. The instructions are to move the emails into these folders using the webmail. Does it mess with the training if I put a forwarded email in to these folders. Let me explain what I want to do:
        1. set up another email called spam@example.com
        2. spam emails i get in myuser@example.com I will forward to spam@
        3. every so often i will log into the webmail for spam@ and then move them into the training folders.
    • Question 2
      • The training I do on one domain, is this stored in a file so I can copy this training to other domain?
      • The file name and location would be nice.
    • Question 3
      • Can you give me a link to documentation from cPanel about spam training so i can look further at it?
    • Answer
      • In regards to your first question, forwarding messages completely alters the e-mail headers and various sections of the e-mail that may interfere with proper training. Rather than identify incoming spam mail, SpamAssassin may begin to think forwarded mail is spam, thus automatically marking all forwarded mail you receive as spam. Training data is shared across entire cPanel accounts rather than domains or individual e-mail users. We can add the training folders to myuser@example.com and then you simply move the spam/ham messages into their respective folders via webmail or IMAP. Afterwards, we can train using this data and that training data will be used for all domains and all e-mail accounts under that cPanel account.
      • If you would like to copy training data to other domains NOT on the same cPanel account, you will need to copy the two files [bayes_seen] and [bayes_toks] from the SpamAssassin directory within the cPanel account. For example, the account [lancast] has it's training data stored in following two files: 
        /home/example/.spamassassin/bayes_seen
/home/example/.spamassassin/bayes_toks
      • These files can be copied and moved to other cPanel accounts to share training data.
      • Unfortunately, cPanel does not offer any direct ability to train SpamAssassin, and as such there is little documentation on the topic:
    • One last question:
      • Q:
        • If i use the inbuilt cPanel forwarding feature this should put a copy of the email in another mailbox without altering it so i can then use that spare account via webmail to move spam into the spam folders without affecting my normal work flow.
      • A:
        • As mentioned previously, we do not recommend setting up a forwarder to send a copy of the messages to another inbox and use the spare inbox to train SpamAssassin.
        • This does alter the message as the message source is now originating from an email account on the server and not the original recipient.
        • The simplest way to fill up your SpamAssassin training folders without affecting your work flow would be to copy the messages from your inbox into the designated SpamAssassin training folders(.SPAM-TRAIN and .HAM-TRAIN).
        • This way you still have the original messages in the folders they were originally in.
      • Q:
        • I am trying to asertain if a cpanel forwarder is the same as a normal email forward. {see image}. I thought that cpanel just made an exact copy of the email message and effectively copied it and not forwarded it in the traditional sense.
        • I am aware now that using normal email forwarding will alter the header.
        • Your method is simple assuming all i use is imap. I am a pop3 person using outlook.
      • A:
        • A cPanel forwarder is still considered a forwarder where the message headers are altered.
        • If you have any further questions or concerns, please let us know and we would be happy to assist.
  • spamtrap
    • add information on this specific feature, is it on by default? can I jst do normal forwarding to it or must it be internal forwarding?
  • Diagnostics
    • EICAR Test File | Trend Micro - The European Institute for Computer Antivirus Research (EICAR) has developed a test virus to test your antivirus appliance. This script is an inert text file. The binary pattern is included in the virus pattern file from most antivirus vendors. The test virus is not a virus and does not contain any program code.
    • I keep getting spam - Virtualmin - Virtualmin Community
      • You need to look in the log to know what’s going on. SpamAssassin mostly works without any user involvement. It can be trained, but it includes a variety of rules by default.
        • Look in the journal for the postfix unit (journalctl -u postfix) to make sure mail is being passed to procmail-wrapper, and then check the procmail.log for whether it’s being processed through SpamAssassin.
        • Then look at the headers of a received mail to see what spam rating it has.
        • URIBL_BLOCKED,URIBL_DBL_BLOCKED_OPENDNS
          • it's referring to the dns server you're using not being allowed to do an RBL request to the the RBL servers.
          • Most RBL servers use a "free for some" method, where as long as a given DNS server isn't doing too many requests, it's allowed. But for a dns server that is too busy, (eg: 8.8.8.8 is very busy), it will be blocked from doing RBL queries, since it no longer qualifies as the "Free for some" method, and would then fall under the category where payment is required to do that volume of RBL queries.
    • Spamassassin (via procmail) is not checking for Spam, Viruses - Virtualmin - Virtualmin Community
      • This includes diagnostics commands as part of this problem work through.
    • Why was a message marked as spam
      • How to find SpamAssassin scan results – cPanel - When SpamAssassin scans an email the results are saved to the /var/log/maillog file. This can be used to determine what rules are being triggered by the message. 
        tail -f /var/log/maillog | grep spamd
      • How can I check why SpamAssassin applied a particular score? – cPanel
        • Some email messages are flagged or rejected as spam, but I'm not sure why. Can I check how SpamAssassin is applying this score? 
          ## You can use the following command to read the rules applied.
su cpaneleximscanner -s /bin/bash -c '/usr/local/cpanel/3rdparty/perl/536/bin/spamassassin -D < /path/to/message'

## You can also use spamc to check by running the following command:
/usr/local/cpanel/3rdparty/bin/spamc < /path/to/message

Please note, that you will need to replace /path/to/message with the full path to the message to scan.
      • How to find the descriptions of SpamAssassin rules to help understand why a message was marked as spam – cPanel - Spam Assassin evaluates a message and assigns it a score to determine whether or not to consider the message spam. It performs the evaluation of the message based on preconfigured rules that tell it what to look for, and what score to apply to the message based on the results of the tests defined in the rule..
  • Troubleshooting
    • Some settings in Webmin, Virtualmin and Usermin do not appear to be functioning as (I) expected - Usermin - Virtualmin Community
      • Summary of issues
        • Default settings for the SpamAssassin ‘required_score’ do not display changes from current overriding settings
        • Changing the accessibility of the SpamAssassin module in Usermin only hides access to the tool but does not change any previously saved configuration which could result in unexpected behaviours for users.
          User prefs file needs to be deleted or possibly renamed if it might need to be reused if access is granted later.
        • The Spam and Virus Scanning dialog under Virtualmin contains features that do not actually pertain to Spam specifically but instead to Filtering tools which can also result in unexpected behaviours and confusion for users.
      • First Issue
        • SpamAssassin reads its configuration from many places in the following order: 
          1. /usr/share/spamassassin
2. /etc/spamassassin
3. /etc/mail/spamassassin (which is a symlink to the previous directory)
4. /etc/webmin/virtual-server/spam/[vm_id] (some files are symlinks, but virtualmin.cf is editable in the UI)
5. /home/[domain]/homes/[mailboxes]/.spamassassin/user_prefs
        • Actually you set the score value in different places this is the reason you get a different behavior. My advice is to set the values for a virtual server in virtualmin.cf, for all mailboxes or per mailbox in user_prefs. Please do your own test by changing the scores in the local.cf, virtualmin.cf, user_prefs files, one by one, and after a change send an email to yourself. Check the header for score number to understand what config file was loaded by SA.
      • Second Issue
        • The option Allow mailbox users to create mail filters has nothing to do with SpamAssassin. Initially I was misled because it is in the SpamAssassin/ClamAV section. However, this option allows you to filter messages using Procmail. Basically, you create a .procmail file in the mailbox and filter the messages based on certain conditions. Unfortunately, this feature is very little addressed, although it has been in Virtualmin for a long time. There aren’t even any examples. I think the option should be changed and the word Procmail introduced, to be clearer. I know the tooltip is there for a purpose, but a word put there can solve the confusion even for an advanced user like me.
    • Why is mail delivery folder different between 2 virtual servers? - Help! (Home for newbies) - Virtualmin Community
      • If you connect an email client software like Outlook, Outlook Express, Apple Mail amongst others, these client software will create and use their own folders which are named as per their own conventions. It is therefore quite common to find not only junk and spam folders but also Sent and Sent Items folders when multiple email clients are used to access a mailbox.
    • no .spamassassin folder in homefolder on user creation - Virtualmin - Virtualmin Community
      • I’ve setup a Virtualmin / LDAP System, which saves the SpamAssassin rules in LDAP Database. All works.
      • But on “user creation / email address creation” no .spamassassin folder will be created in it’s homefolder.
      • Work-Around for me is to login with usermin and just open SpamAssassin Mail Filter Option under the ne account. Afterwards the .spamassassin folder will be created.
      • It’s spamc. I’ve read the help this for, yeah and it says, that only spamassassin standalone can manage per user/domain settings.
    • Postfix doesn't pass mails through SpamAssassin anymore - Webmin - Virtualmin Community 
      • mailbox_command = procmail -a "$EXTENSION"
        • This is why. That’s not the configuration we use in Virtualmin. You need procmail to be able to switch user to the receiving user in order to process personal procmailrc files.
        • You could run it with regular procmail, but you’d have to configure a system-wide procmailrc that sends mail through whatever processing you want to do (you lose Virtualmin-managed per-user filters, autoresponders, etc. in this case).

SpamAssassin Addons

Razor Spam Detector
DCC Plugin for SpamAssassin
Pyzor

Rspamd (not in Virtualmin officially)

DNS

  • Reset the DNS zone (There are couple of ways to reset the DNS zone)
    1. Virtualmin --> Limits and Validation --> Validate Virtual Servers --> Reset Features
      • Virtualserver to reset: Select the relevant Virtual Server
      • Features to rest: DNS domain
    2. Command Line 
      virtualmin reset-feature --domain example.com --dns
    3. Virtualmin --> DNS Settings --> DNS Records --> Reset DNS Zone
      • This button has not been added yet, but should be shortly.
  • Why is there a 5 added at the begining of the MX record.
    • Virtualmin --> DNS Settings --> DNS Records
    • The 5 is supposed to be there, it represents the Mail server priority
  • After one week my DNS still has not fully propagated, why?
  • What DNSSEC algorithm to use?
    • = Algorithm 13 (ECDSA Curve P-256 with SHA-256) (ECDSAP256SHA256)
    • RFC 8624 - Algorithm Implementation Requirements and Usage Guidance for DNSSEC
      • The DNSSEC protocol makes use of various cryptographic algorithms in order to provide authentication of DNS data and proof of nonexistence. To ensure interoperability between DNS resolvers and DNS authoritative servers, it is necessary to specify a set of algorithm implementation requirements and usage guidelines to ensure that there is at least one algorithm that all implementations support. This document defines the current algorithm implementation requirements and usage guidance for DNSSEC. This document obsoletes RFC 6944.
      • RSASHA1 and RSASHA1-NSEC3-SHA1 are widely deployed, although the zones deploying it are recommended to switch to ECDSAP256SHA256 as there is an industry-wide trend to move to elliptic curve cryptography. RSASHA1 does not support NSEC3. RSASHA1-NSEC3-SHA1 can be used with or without NSEC3.
      • Has a chart showing what to use and why.
    • DNSSEC specification recommends not signing with DNSSEC algorithm 7 (RSASHA1NSEC3SHA1) · Issue #1953 · mail-in-a-box/mailinabox · GitHub
      • Cloudflare DNSSEC is now exclusively algorithm 13.
      • Algorithm 13 (ECDSA Curve P-256 with SHA-256) (ECDSAP256SHA256)
      • Has a chart and is a discussion about what algorithm to use.
  • No delegation NS records were detected in the parent zone (DNSSEC) 
    quantumwarp.com to wordpress.quantumwarp.com: No delegation NS records were detected in the parent zone (quantumwarp.com). This results in an NXDOMAIN response to a DS query (for DNSSEC), even if the parent servers are authoritative for the child. (31.125.252.137, UDP_-_EDNS0_4096_D_KN)
    • Notes
      • This error can be frustrating but easy to fix.
      • You do not need to have different Nameservers for each domain and sub-domain in the chain.
      • You do not need to have all of youer domains in the same zone file, but you can do if you want.
      • If there is a break in the DNSSEC chain then you will always get a NXDOMAIN response.
    • Causes
      1. The required DS and NS records in the parent domain are missing or badly formatted.
      2. If all the records are set correctly, then it is just a case of waiting because some of these records need updating at the registrar. Virtualmin's default TTL is 3600s (1 hour). For me after making the changes it took about 2 hours for my domains to become resolvable. Usually it will take between a couple of hours and in extreme cases 48 hours, but any more means you ahve an issue in your DNSSEC chain and that will need fixing.
    • Links
      • linux - Error adding DS records for my subdomain to the zone file of parent domain - using bind - Server Fault
        • DS records are only used as part of delegations between zones, ie side by side with the NS records that define such a delegation.
        • If you have for example the zone example.com and just add records for foo.example.com or foo.bar.example.com to this zone that is already covered as it is part of the same zone.
        • However, if you delegate eg sub.example.com so that this is a separate zone, you would have BOTH NS and DS records for sub.example.com in the example.com zone.
        • I'm not sure which of the cases above this question describes, but either you are missing the NS records for the delegation of the new zone or you are trying to add superfluous DS records "within" a zone.
    • DNSSEC Tools
      • DNSViz | A DNS visualization tool - DNSViz is a tool for visualizing the status of a DNS zone. It was designed as a resource for understanding and troubleshooting deployment of the DNS Security Extensions (DNSSEC). It provides a visual analysis of the DNSSEC authentication chain for a domain name and its resolution path in the DNS namespace, and it lists configuration errors detected by the tool.
      • Documentation | DNSVizl
      • DNSSEC Guide : Common Problems | The DNS Institute - DNS tools, DNS documentation, DNS consulting, DNS analysis.
      • DNSSEC Debugger - The DNSSEC Debugger from VeriSign Labs is an on-line tool to assist with diagnosing problems with DNSSEC-signed names and zones.
  • Do Virtual Servers need nameservers and corresponding NS and A records for them?
    • Scenarios
      1. You are not managing DNS for the domain on Virtualmin
        • Neither the NS or A records are not needed because they will never get used.
      2. You are managing the DNS on Virtualmin, and the GLUE records at the registar are pointing to your Virtualmin server (i.e. you say the nameservers are here, child nameservers)
        • Both NS and A records are needed
      3. You are managing the DNS on Virtualmin, but the nameservers are on a different domain
        • Only NS records are needed
    • Explanation
      • All domains must have at least one nameserver. It is preferred to have 2 NS records but most configurations point both to the same server.
      • NS records give the authoritive domain to perfom DNS requests for this domain.
      • Nameservers do not have to be on the same domain.
      • An NS record always needs a corresponding A record and these A records are always on the same domain as the nameserver.
      • If a defined NS record is for a remote nameserer/domain, then there is no need for an A record on the that Virtual Server because the required A record is on the remote server.
      • DNS Zones can contain just 1 domain or that domain and subdomains. Each zone must have at least 1 nameserver defined and it is best practice to have 2.
    • Notes
      • Client Virtual Servers - Do they need name server (NS records) - Virtualmin - Virtualmin Community
        • The A record points to the IP of the NS record
        • The NS record points to the DNS Server
        • The NS record although not showing an IP within the entry still has to point at an IP. This gets done with an A record.
        • This A record needs to be on the zone record that is hosting the NS.domain.tld
        • If they are hosting their own NS? where is the IP coming from if the A record is not created? Therefore the A record needs to be included in their Zone.
      • domain name system - DNS subdomain (child) NS records - Server Fault
        • The authoritative NS records reside inside the zone itself (and provided in ANSWER section when the authoritative server is queried), just like all other records that are part of that zone.
        • To be able to traverse the tree, referral/delegation/authority information (NS and any glue A/AAAA records necessary) is also added to the parent zone.
        • This information, however, is not treated as the "real answer", the answer lacks the AA (authoritative answer) flag and the NS records are in the AUTHORITY section to indicate that this is just information on who has the actual answer.
        • One implication of this is that if you do a direct lookup of NS records you will follow this referral and query the authoritative server despite having just seen what should be the same information.
        • There is more to this answer....
      • 10 Child Domain DNS Best Practices - CLIMB - DNS is a critical component of any child domain, and there are a number of best practices that should be followed in order to ensure its security and stability. In this article, we'll cover 10 of the most important DNS best practices for child domains.
      • Parent Zones and Child Zones (Cut Nodes) in Managed DNS | Dyn Help Center
      • domain name system - What type of DNS record is needed to make a subdomain? - Server Fault
      • What Is A DNS NS Record? A Complete Guide To NS Records
        • A DNS NS record is a type of resource record in the Domain Name System (DNS) used to specify nameservers for a domain.
        • DNS NS records identify the authoritative name servers for a zone.
        • Every zone must have at least one entry that identifies the name servers responsible for the domain. The availability of a zone can be increased by using two or more such records. If the first name server is unavailable, the zone will still be accessible via another server.
      • domain name system - Clarification of why DNS zone files require NS records - Server Fault
      • The NS record | NsLookup.io - DNS NS records specify the authoritative name server for a domain. Learn how name server record works and how to configure them.
      • How to Configure DNS Nameservers with cPanel | cPanel
        • In this article, we’re going to explore some essential DNS concepts and then show you how to configure a custom server with cPanel.
        • A private nameserver lets hosting providers give their customers a branded address such as ns1.mycompany.com.
        • Another benefit is that you control the domain. If you move to another server hosting platform, your domain comes too, and your clients don’t have to change their configuration.
      • domain name system - DNS A vs NS record - Server Fault - I'm trying to understand DNS a bit better, but I still don't get A and NS records completely.
      • networking - How are the NS records resolved? - Server Fault - This give the full workflow of a DNS request and how NS and A records play a part in that.
      • Is it possible to set up a sub-domain to point to a different name-server? - #5 by Joe - Virtualmin - Virtualmin Community
        • A zone (when you register a domain, a zone is created for that domain) has NS records, and they delegate authority to DNS servers. That kinda feels like a domain pointing to name servers, to me.
        • Your registrar is responsible for those NS records…they’re the glue records in your domain zone. They can point to your name servers managed by Virtualmin, or they can point to the registrars DNS servers, or some other DNS servers. When you do a whois on a domain (or dig with appropriate options), it’ll show those NS records: 
          $ whois virtualmin.com|grep 'Name Server'
Name Server: ns1.virtualmin.com
Name Server: ns2.virtualmin.com
        • Those two name servers can then delegate (point to) other name servers to be responsible for subdomain names under that domain name. Or all the names and subdomains can be served by those DNS servers without delegating anything.

Cron / Cronjobs

These are very useful for automating tasks

  • How to setup a cron job – Virtualmin - This tutorial covers how to setup a Cron job. Cron is a service for executing scheduled commands.
  • Located at
    • (System) Webmin --> System --> Scheduled Cron Jobs
    • (User) Virtualmin --> Webmin Modules --> Scheduled Cron Jobs

Software Package Management

  • Software Package Updates and Software Packages are different
    1. Software Package Update
      • Webmin --> System --> Software Package Update
      • This handles your standard repository tasks as if your were using apt-get on the command line.
      • This is what you would class as the package manager if anyone asks.
      • Software Package Updates | Webmin
        • About The Software Package Updates module shows available updates and provides for actual updating.
        • It cannot remove packages.
    2. Software Packages
      • Webmin --> System --> Software Packages
      • This is only concerned with local operations such as:
        • Manually installing a package.
        • Automatically upgrading the install packages.
        • Listing installed packages.
        • Not 100% of this feature's role.
      • Software Packages | Webmin
        • This chapter covers the installation and management of software on your system using packages.
        • It also covers the differences between the various Unix package formats, such as RPM, DPKG and Solaris.
        • Introduction to packages All Linux systems use some kind of software packaging system to simplify the process of installing and removing programs.
        • A package is a collection of commands, configuration files, man pages, shared libraries and other files that are associated with a single program like Apache Webserver or Postfix Mail Server, combined into a single package file.
        • The Software Packages module can be used to install/remove other packages.
  • PostgreSQL
    • Check to see if PostgreSQL is installed
    • Uninstall PostgreSQL
    • Disable PostgreSQL
      • Uninstalling PostgreSQL? - Help! (Home for newbies) - Virtualmin Community
        1. first make sure Virtualmin isn’t using it
          • Virtualmin --> System Settings --> Features and Plugins --> "PostgreSQL database": uncheck
        2. Next, you can prevent Postgres from loading on startup by going into
          • Webmin --> System --> Bootup and Shutdown --> Postgresql --> Start at boot: No
            • This might not be present if the service is not installed or has an init script.
    • PostgreSQL removed from the default installation
      • Postgresql won't enable in virtualmin - Help! (Home for newbies) - Virtualmin Community
        • Of course it’ll let you install it! It’s just a regular package from your OS vendor. Once installed, you can enable it in Virtualmin.
        • We removed pg from the default installation because so few people use it (I prefer it slightly, but there’s not much we can do to change the vastly larger preference for MySQL/Mariadb among the projects in the Install Scripts (Manage Web Apps) and in the web dev community in general).
        • Use your system package manager to install postgresql and postgresql-server packages (probably, you haven’t mentioned your distro and version, but I think that’s the right name on all distros we support). You can use the Webmin Software Packages module to do it or do it from the command line. You’ll also probably want the php pg driver packages, or the relevant drivers for the language(s) you’ll be developing with.
        • Once that’s done, you can use: 
          # virtualmin config-system PostgreSQL
        • To do some minor initial configuration (this may not be necessary, depending on your distro/version). Then you can either enable postgresql-server (systemctl enable postgresql-server) or re-run the Virtualmin post installation wizard to enable it.
      • Virtualmin 6.2.0 - ubuntu 20.04 PostgreSql - Virtualmin - Virtualmin Community
        • Q: The latest version of virtualmin apparently does not install the necessary Postgres packages, since the moment to ask if mysql is installed and also PostgreSql also advances installing Mysql, if PostgreSql is also selected it sends an error similar to when Huge Mysql is selected
        • A:
          • Try running 
            apt-get install postgresql postgresql-client libdbd-pg-perl libpg-perl
          • PostgreSQL is not installed by default on Ubuntu 18.04 or 20.04. It’s listed as Suggests: in the package, but most people don’t have suggested packages enabled. I think I wanted to reduce the initial install size and complexity, and very few of our users use PostgreSQL (despite it being superior to MySQL/Mariadb in some regards), so it needs to be installed if you want to use PostgreSQL.
          • It was an intentional change to remove it from the default install, but you’re not clear about what problem you’re seeing in the setup wizard? Is it offering PostgreSQL options? It shouldn’t if the postgresql packages aren’t installed…that’d be a bug, but not one I’ve seen.
  • Installing REDIS
    • Redis - official way of installing and configuring in Virtualmin - Virtualmin - Virtualmin Community
      • There is no official way. Use whatever is appropriate for your distro and version. Your operating system is still the same, Virtualmin is just managing some parts of it.
      • There is no Webmin or Virtualmin module for Redis that I know of (certainly none from us, though maybe someone else has implemented one, but I don’t know of one); it hasn’t come up much. One could certainly build one without a lot of work.
      • Virtualmin is not your OS. Virtualmin only cares about the packages it manages, and the packages it manages are installed using your operating system’s package manager, and using the OS standard repositories whenever possible. Virtualmin itself is installed using your operating system package manager (apt-get/dpkg on Ubuntu).
      • Q: Now, if I start installing custom php modules or even building them from source, how will it affect Virtualmin?
      • A:
        • As long as you don’t break PHP, it doesn’t matter. We don’t even use PHP. We just configure it for you, we don’t depend on it in any way.
        • If you can install a package using the OS package manager and standard repositories you should do so.
        • Third party repositories should be used with caution, only when necessary, and only after testing.
        • redis and php-redis packages are available in the Ubuntu repositories, I’d recommend you use those. Installing from source should be a last resort (and, I never allow from-source installs on my production servers). But, that has nothing to do with Virtualmin.
        • That’s me offering you advice based on my decades of systems management. Virtualmin don’t care about Redis.
      • Many thanks, it is clearer now. Key takeaways:
        • If you can install a package using the OS package manager and standard repositories you should do so.
        • Installing from source should be a last resort
        • redis is Redis, php-redis is PHP bindings for Redis. They have different and unconnected versions.
      • You need to install these packages:
        • redis
        • php-redis
    • Complete Guide to Redis PHP - GeeksforGeeks - A Computer Science portal for geeks. It contains well written, well thought and well explained computer science and programming articles, quizzes and practice/competitive programming/company interview Questions.
    • Redis installation from within Virtualmin - Blue Skies - Virtualmin Community - Charles outlines the simple steps to install REDIS.
  • Installing Memcache
  • Installing OPCache
  • Install ImageMagick
  • Install GD module (Software Package Updates)
    • Webmin --> System --> Software Package Updates
    • States to display: Only new
    • Find packages matching:
      • php-gd
        • This will install GD for the system default PHP version.
      • php8.3-gd
        • This will install for PHP version 8.3
  • Install GD module (manually)
  • How to use the 'Software Package Updates' module to install PHP extensions
    • Webmin --> System --> Software Package Updates --> Package Updates --> Only new
    • Notes
      • Each version of PHP will needs it's own version of the extensions installing. Be aware that some extensions might not be available on a particular version of PHP.
      • You use this module to search for the packages you want, tick them and then install them which is very straight forward.
      • All of this can be done with the command prompt.
      • If you cannot see a package, it is probably already installed.
      • The trick to installing all of the packages you want is how you search for them.
      • You cannot use wildcards in this module or at least not that I have got to work.
      • Your OS will have a default version of PHP installed and if you do not specify a PHP version the OS will convert the generic term to the OS specific version of PHP before makeing the request.
      • The search does a %TERM% style search.
    • Search terms and their outcomes
      • php-curl
        • This will find the php-curl package only as it is a very specific search term.
        • My OS default PHP is 8.2 so it will install the php8.2-curl package.
        • Defaul OS PHP Packages (PHP 8.2)
      • php8.1-curl
        • This will find the php8.1-curl package only as it is a very specific search term.
      • -curl
        • This will find all of the curl packages, and more.
      • curl
        • This will find all of the curl packages, and more but is not as specific as the command above.

Software Package Configuration and Usage

phpMyAdmin

Security

General

  • You can restrict access to Webmin and Usermin by IP or hostnames.
  • You can restrict access to Webmin and Usermin with root account (or other account) by using IP or hostnames.
  • Settings for best security - Help! (Home for newbies) - Virtualmin Community
    • For starters, to get an inherently secure system, it’s recommended to use a Grade-A supported OS, installing no packages besides SSH, and using the Virtualmin installer script to get your web hosting software in place. Virtualmin configures the services, as securely as you can get without being an employee at one of the aforementioned firms.
    • Most security issues come from buggy or incorrectly configured web software, and not from the services itself.
    • My suggestion would be: First, turn off “root login with password” in SSH. Set it to “with RSA key only”. That will prevent brute force attacks on the root account, because no brute force attack in this world can work out an RSA key (of sufficient length).
    • In Virtualmin, you’d still use the root user and their regular password (make it securely long). Brute-force attacks on Webmin are very rare, since it’s by far not as widespread as SSH.
    • If you want extra security, set up a VPN (OpenVPN suggested) and open port 22 and 10000 only for VPN connections.
    • For optimum security it is always a good idea to go through some security/hardening check lists.
    • A few items which rank high on my list of security measures include, “firewall hardening”, “disabling FTP (and other services not used) in favour of SFTP”, “disabling password authentication for root”, and installing a good “intrusion detection system”.
    • We have been using OSSEC for our primary OS-level intrusion detection system for a few years now, and it has saved us sleepless nights because of it’s highly customizable ruleset, and the proactive measures it takes against hackers and other malicious activity. OSSEC also if configured will send out an email to you including all items which may be a security threat, or that you should know about including login attempts, file changes, etc. When you consider what OSSEC and similar software does, it makes administrating lots of machines less of a headache, and increases uptime by pointing out threats, and taking proactive measures.
    • CSF/LFD: Watches, among lots of other things, logs for login failures and blocks the offending IP via iptables. Also watches for modified system files, can detect port floods, use blacklists to block known hacker nets, limit connection count per source IP, and other stuff.
    • LOGCHECK: Scan configurable log files and reports all lines it doesn’t know (configurable via regular expressions, comes with a pre-made set of rules) via email
    • LMD: Linux Malware Detect, a malware scanner specifically for bad web software. Uses the ClamAV engine for scanning.
    • To mitigate the brute force of Virtualmin using the root user, you could make sure you tighten the host blocking options:
      • Webmin --> Webmin Configuration --> Authentication
      • I would leave “Block users with more than” and “Lock users with failed logins” otherwise you might get locked out of root access as I am not aware of any white listing option. Perhaps turn up the time a host is blocked for invalid login attempts? You could also change the port that is used to access Virtualmin but that isn’t really security (in my opinion, security through obscurity doesn’t do much except slow down a determined attacker).
    • Lawk - This is what I do after a clean virtualmin install on a minimal OS install:
      • Disable root login by SSH, instead I use a regular user to login and then “su” for root. I guess you could also use keys.
      • Enable the iptables firewall in webmin to only allow the hosting ports.
      • Install & configure fail2ban, enable it not only for SSH, but PAM, postfix, proftpd, dovecot, perhaps others, in more recent versions there will be a Webmin jail too so you can use that out of the box.
      • Create a Virtual Server with a domain and make sure SSL is enabled as a feature.
      • Get the Let’s Crypt Certificates in “manage SSl” through virtualmin server management. This has the benefit of enabling SSL in those applications…
      • BUT I always change the protocols and ciphers to something along the lines of: https://cipherli.st/
      • So that only TLS 1.2 is used.
      • You can then add HSTS to Apache. (careful though that auto renewal works for the certs and that you are not using self-signed).
      • You then get the A+ rating on Qualys.
      • Then you can always run stuff like Nessus & Netsparker to scan for anything you might have missed of known vulnerabilities.
      • etsparker can scan your webapps for problems in php and so on.
  • SSH Server | Webmin - A worthy read.
  • Mod_security and/or firewall for new setup - Virtualmin - Virtualmin Community
    1. configure mail rate limiting to limit damage that can be done by spammers who gain unauthorised access to user accounts
    2. configure fail2ban to thwart brute force attacks
    3. use only php-fpm as execution mode on all virtual servers to keep the system isolated from virtual servers that will be compromised
  • Suggestions of a New Noob - Blue Skies - Virtualmin Community
    • mod_security with recent CRS rules provide functionality in a similar vein to mod_evasive. I’ve opted to use those on my deployments, rather than using both tools.
    • This is the best explanation for that I could find with a quick search: apache - Apache2 mod_evasive vs mod_security with OWASP crs when protecting against DDOS? - Stack Overflow
    • There are probably better docs for using CRS rules, though.
    • At this time I don’t see any compelling reason to use both, and one could create fail2ban rules to watch for mod_security actions, too, if you wanted to make the layer 7 blocking decisions at layer 4 instead (which could likely provide a small benefit in severe DDoS situations).
  • IDS (Intrusion Detection System) - #4 by happycoding - General Discussion - Virtualmin Community
    • Intrusion Detection System (IDS):
      • IDS monitors and analyzes network or system activities for signs of malicious behavior or security policy violations.
      • It operates in a detection-only mode, meaning it identifies and alerts about potential threats but does not take direct action to prevent them.
      • IDS can be network-based or host-based, depending on whether it monitors network traffic or activities on individual systems.
    • Intrusion Prevention System (IPS):
      • IPS, on the other hand, not only detects malicious activities but also takes proactive measures to prevent them.
      • It can block or prevent malicious activities in real-time by actively intervening in the network or system processes.
      • Fail2Ban falls into this category because it actively responds to detected malicious behavior by blocking IP addresses, thereby preventing further unauthorized access.
    • Fail2ban:
      • Fail2Ban is specifically designed to protect against unauthorized access attempts by monitoring log files for patterns indicative of a potential security threat, such as repeated failed login attempts.
      • When it detects such patterns, it can automatically update firewall rules to block the source IP addresses of the potential attackers.
      • While Fail2Ban is not a full-fledged IDS, it provides a level of intrusion prevention by responding to specific events that may indicate malicious intent.
  • Apache Hardened Web Server - Rocky Linux Documentation - Whether you are hosting many websites for customers or a single important website for your business, hardening your web server will give you peace of mind at the expense of a little more up-front work for the administrator.

fail2ban vs CSF

  • My Thoughts
    Use the default Fail2Ban and FirewallD setup unless you know why you want CSF.
    • FirewallD + Fail2Ban are good to go straight out of the box and will cover most peoples needs. If you want more options and control you can use CSF+LFD.
    • FirewallD and Fail2Ban modules are made by Webmin so will get updated by the team, whereas CSF is controlled by the folks at ConfigServer.
    • FirewallD is GUI to iptables, configured with multiple zones and with a simple GUI makes this a great choice. It does what it says on the tin.
    • Fail2Ban is used in conjuction with FirewallD and is a well tested IDS and brute force login blocker.
    • ConfigServer Security Firewall  (CSF) has a firewall and a login daemon (LFD) to provide a great level of security and options. This software requires more setup and management but you can get more out of it.
    • Comodo WAF is a set of ModSec rules (OWASP) using the mod_modsec apache module to provide a deeper level of protection. This installtion can be tricky. The virtualmin team are hoping to brinf ModSec to Virtualmin Pro 8.
  • ConfigServer Security & Firewall (csf) - Third Party News - Virtualmin Community
    • Q:
      • Has anyone used ConfigServer Security & Firewall (CSF) with Virtualmin. It was recommended to me and on its website it says it has a module for Webmin.
      • Is it worth using? What are the pros and cons? Is it more or less effective than the controls in VM? Would be grateful for +ve and -ve experiences.
    • A (Ilia):
      • ConfigServer Security & Firewall (csf) has a great support with Virtualmin and default Authentic Theme, simply because I was personally using it.
      • CSF is a great piece of software but it depends on your needs.
      • Nevertheless, standard Virtualmin setup with FirewallD + fail2ban does all the job pretty well and is more than enough usually.
      • Yes, neither firewalld nor fail2ban can be used alongside with CSF. CSF has its own implementation of login failure daemon called lfd.
      • Also, you shouldn’t worry about firewalld and fail2ban, as CSF installer would take care disabling them for you.
  • Fail2Ban versus CSF? | vpsBoard
    • Q: Which one do you think is better? I have lot of experience with CSF on cpanel servers but not on a server without it. I've never used fail2ban before. Which one would be better for a vps that has no control panel?
    • A:
      • I would say that the two have different applications: One (CSF) is a firewall frontend with Intrusion Detection Service (LFD) and the other is a plain Intrusion Detection Service (fail2ban). If you do not need the firewall part of CSF, then I would go with fail2ban
      • Base functionality for the average user, fail2ban and LFD will be no noticeable difference. Of course, CSF is a nice easy way to fine tune iptables for the average user and for that I highly recommend it.
  • which is the best protection? fail2ban or CSF - Vesta Control Panel - Forum
    • Two different purposes. CSF is Firewall and fail2ban is a plain Intrusion Detection Service.
    • CSF is actually a firewall which includes a brute force protection daemon, very similar to fail2ban. I think this is what prompted the original poster's question.
    • From the website - Login Failure Daemon (lfd)
      • To complement the ConfigServer Firewall (csf), we have developed a Login Failure Daemon (lfd) process that runs all the time and periodically (every X seconds) scans the latest log file entries for login attempts against your server that continually fail within a short period of time. Such attempts are often called "Brute-force attacks" and the daemon process responds very quickly to such patterns and blocks offending IP's quickly. Other similar products run every x minutes via cron and as such often miss break-in attempts until after they've finished, our daemon eliminates such long waits and makes it much more effective at performing its task.
  • My firewalld isn't working - what is the correct FirewallBackend? Please check yours for me? - #22 by jotst - Help! (Home for newbies) - Virtualmin Community
    • Illia
      • FirewallD is just a front end for iptables/nftables. It does nothing on its own.
    • MrPete:
      • Here’s my new understanding of the reality:
        • FirewallD (and firewall-cmd) is not a firewall at all. It is a UI for a backend firewall, either the older iptables or newer nftables packet filters, and other associated bits.
        • iptables refers to two entirely separate things (managed by the netfilter.org 2 project)
          • a kernel packet filtering technology (being replaced by the nftables packet filter)
          • the iptables firewall definition command utility (nft is the equivalent for nftables architecture)
        • Because the actual packet filters are built into the kernel, there’s no visible iptables or nftables process.
      • So in simple terms:
        • FirewallD is a front end that currently uses iptables as its backend.
    • Joe
      • You’re right, the Webmin Firewalld module is pretty limited (and Firewalld is kinda confusing, I have to read the docs every time I touch anything…I wish it weren’t the best option, but it pretty much is, at least for our needs and our users most common use cases).
    • Joe
      • There is very good support for CSF in Virtualmin/Webmin and Authentic Theme. But, I don’t like that sort of firewall and don’t recommend it on servers. It’s very easy to get bogged down in the minutiae of complicated rules that don’t make sense on a web server. But, Ilia likes CSF, so there’s good support for it.
      • But, Ilia has been doing a lot of work on the Firewalld module, so it’s going to get some upgrades in the next Webmin release.
    • MrPete
      • Q:
        • CSF Firewall comes with a feature called Login Failure Blocking if you do not want to use permanent blocking.
      • A:
        • Read what you quoted about CSF Login Failure Blocking: it’s either for a specific time frame, or permanent.
        • Fail2Ban is far more flexible and robust:
          • It can monitor ANY type of error found in ANY log file
          • The block can be set for ANY number of failures
          • The initial time can be ANY amount of time
        • And, in the upcoming 0.11 release (available “out there” but coming soon to Virtualmin), the block can grow exponentially with each failure, which is very very nice.
        • I had a tiny server suffering from a couple dozen attacks per second. Turned on exponential-growth blocking (when really bad, I let it grow to a one-month block :wink: ) and everything worked Just Fine.
  • Firewall or other security - Help! (Home for newbies) - Virtualmin Community
    • So, you’ve got a couple of obvious options. One would be to setup iptables (more flexible and, I think, more useful, on servers, but also more complicated), the other would be to start firewalld. Webmin has a module for either; there’s also a CSF module for Webmin, but that may be overkill for your needs. I usually use iptables, because I know it really well, and it is flexible and powerful enough for everything I need.
    • Firewalld is the new management service used, by default, in CentOS 7 and recent versions of Fedora. It is integrated with systemd, which allows it to dynamically apply rules based on what’s running, and the network your system is connected to (e.g. if you have a wired network at work and a wifi network at home, the firewall can act differently in either case). But, for servers, the additional features are pretty much extraneous and may even get in the way. For a server, you mostly just want to say, “Open these ports, and leave them open forever, because I have services running on them.”
    • I’m surprised firewalld isn’t already running; I though it was on by default on a CentOS 7 system. The fact that it’s not running might mean it didn’t get new rules added when Virtualmin was installed. Our installation detects which firewall you have (whether iptables or firewalld on CentOS) and inserts the rules in needs for all of the services it manages. You can, of course, customize those rules at any time in the Linux Firewall or Firewalld module.
  • Fail2Ban already banned - #30 by dimgr - Help! (Home for newbies) - Virtualmin Community
    • Ilia
      • Fail2Ban is essentially LFD (Login Failure Daemon) in CSF. It does exactly what Fail2Ban does. There is no need to have Fail2Ban if you are using CSF, period!
      • Q: As an expert in virtualmin, which one do you recommend I leave for better security? I see a lot of attacks on my vps postfix_sals
      • A:
        • Virtualmin isn’t involved, but either option works. LFD in CSF is a more powerful and configurable tool than Fail2Ban. However, both ultimately serve the same purpose—significantly reducing the chances of successful brute force attacks.
        • Postfix and other services will always be “under attack” by bot-nets. This is normal for any server facing the internet.
        • The only thing you should really worry about is making sure every user on your server has a super strong password for each service they use.
    • Joe
      • I think you should stop doing dramatic things because of a minor configuration issue. Installing CSF, which you’ve never used and have no experience with, because fail2ban had one misconfigured jail is absolutely bonkers.
      • So, I think you should probably stick to a default installation until you’ve got some experience before you go off-roading by replacing big chunks of the system with random stuff.
    • Ilia
      • You should stop trying to use LFD alongside Fail2Ban! These are similar tools. The actual bug is in the csf/install.generic.sh script, which disables firewalld but doesn’t disable fail2ban.
      • Long story short—if you use CSF, you shouldn’t use Fail2Ban! Stop making your life more complicated! You don’t need all this micromanagement—who cares who’s trying to brute force your user password? Just set a strong password, and if there is a bug in Postfix, Fail2Ban won’t help you!
  • ConfigServer CSF - #3 by Smedby - Virtualmin - Virtualmin Community
    • I think CSF is great, but I don’t believe it’s necessary to replace the Virtualmin stock FirewallD + Fail2Ban. Essentially, they are equally effective and perform the same functions.
    • I like you can do country blocking with CSF and add quick blocking of IP’s too.
    • CSF has the unpleasant side effect of blocking you from the server, not just the service, you triggered. Not pleasant when doing re,mote admin. Happens too much when someone is trying to set up email too.
    • Quick IP manipulation was the only real reason I went into CFS once it was set up.
    • CSF give a graph of the country its blocking the most.
  • Switch from UFW and fail2ban to CSF – Everything is Broken - Having played with CSF for a while on one server, I've decided I like it more than UFW and fail2ban. It seems much better at blocking mail bruteforce attacks and SSH as a distributed attack.
  • Firewall commands
    • List all firewall rules 
      firewall-cmd --direct --get-all-rules
    • What is the output of the following commands? Do you have iptables package installed? 
      apt list --installed |grep -i tables
which iptables
whereis iptables

Linux Binaries

These are the undelying kernel level drivers that handle blocking and allowing of traffic. All the firewalls are a level up and utliise these conmands.

iptables
  • There is a module for naked iptables/nftables
    • Webmin --> Un-used Modules --> Linux Firewall
  • iptables is just a command-line interface to the packet filtering functionality in netfilter
  • iptables is utilised by many frontends that just configure the iptables to rules to do their bidding.
  • Iptables Tutorial - Beginners Guide to Linux Firewall | Hostinger - Iptables is a powerful firewall tool for Linux. Read our Iptables tutorial and learn everything you need to know to secure your server.
  • Iptables Tutorial: Ultimate Guide to Linux Firewall - Learn all about iptables and Linux firewalls in this ultimate tutorial. Configure iptables and secure your server workloads before a cyber attack strikes.
  • An In-Depth Guide to iptables, the Linux Firewall - Boolean World - The Linux kernel comes with a packet filtering framework named netfilter. It allows you to allow, drop and modify traffic leaving in and out of a system. A tool, iptables builds upon this functionality to provide a powerful firewall, which you can configure by adding rules. In addition, other programs such as fail2ban also use iptables to block attackers. In this article, we’re going to take a look at how iptables works. We’re also going to look at a few examples, which will help you write your own rules.
  • How the Iptables Firewall Works | DigitalOcean - The iptables firewall is a good way to protect your server from unwanted traffic from the internet. in this guide, you will review how Iptables works.
  • A Deep Dive into Iptables and Netfilter Architecture | DigitalOcean - Firewalls are an important tool that can be configured to protect your servers and infrastructure. In the Linux ecosystem, iptables is a widely used firewall tool that works with the kernel’s netfilter packet filtering framework.
  • networking - How can I use iptables on centos 7? - Stack Overflow
nftables
  • There is a module for naked iptables/nftables
    • Webmin --> Un-used Modules --> Linux Firewall
  • Debian 10 Firewalld vs iptables thrashing about - Help! (Home for newbies) - Virtualmin Community
    • Instructions on how to uise nftables instead of iptables. This is a couple of years old so the transition might already of happend, for me it has.
    • linux - Check whether iptables or nftables are in use - Unix & Linux Stack Exchange
    • Why nftables instead of iptables?
      • Starting with Debian 10, iptables is officially deprecated with nftables. With Debian 11 the deprecated goes even further. iptables is now the default on Debian 11. Source at Debian 11 To Further Deprecate IPTables In Favor Of Nftables Plus Promoting Firewalld - Phoronix
      • Starting in August 2020, nftables is included into the Linux Kernel. Which result in potential significant increase in both performance & security.
      • Fail2Ban on Debian 10 has very good support for nftables. With lots of built-in configurations.
    • Notes
      • For those not familiar with nftables. It is the new framework by the Netfilter Project. Which allows you to perform packet filtering (firewalling), NAT, mangling and packet classification.
      • firewalld is a front end management tool for nftables. Think of nftables as the engine. And firewalld as your dashboard.
      • Firewalld “owns” the firewall on the system, and all management should be done using the firewalld commands or the Webmin firewalld module. Attribution to Joe at https://forum.virtualmin.com/t/firewall-iptables-and-firewalld-conflict/58278/5
      • For those not familiar with Backport. It means you get more recent version of packages for Debian.
      • nftables replaces the old popular iptables, ip6tables, arptables and ebtables
  • How to Use nftables | Linode Docs - In this guide you will learn about what nftables is and how it differs from iptables, plus you""ll get a look at how to use and create tables, rules, and chains.
  • nftables - Debian Wiki - nftables is a framework by the Netfilter Project that provides packet filtering, network address translation (NAT) and other packet mangling.

Firewalls

FirewallD
  • Webmin --> Networking --> FirewallD
  • FirewallD is just a front end for iptables/nftables. It does nothing on its own.
  • Cannot delete a rule in FirewallD
    • Webmin --> Networking --> FirewallD --> load any zone --> List FirewallD Rules
    • You see a rule that you don't recognise or want to remove, but there is no option to select or delete. This rule is probably visible in all zones.
    • This 'Direct' rule is created by Fail2Ban and cannot be deleted here.
    • This behaviour is not a bug.
    • The rule can be found here: Webmin --> Networking --> Fail2Ban Instrusion Detector --> Jails Status
    • You can clear the block here or it will probably clear itself in 15 minutes.
Fail2ban
  • eneral
    • Webmin --> Networking --> Fail2Ban Intrusion Detector
    • fail2ban is a login daemon that makes descisions by inspecting the logs and then making changs to the blocing tables using iptables
      fail2ban is run in tandom with the UFW firewall on Ubuntu.
      Fail2ban is a service that uses iptables to automatically drop connections for a pre-defined amount of time from IPs that continuously failed to authenticate to the configured services.
    • GitHub - fail2ban/fail2ban
      • Daemon to ban hosts that cause multiple authentication errors
      • Fail2Ban scans log files like /var/log/auth.log and bans IP addresses conducting too many failed login attempts. It does this by updating system firewall rules to reject new connections from those IP addresses, for a configurable amount of time. Fail2Ban comes out-of-the-box ready to read many standard log files, such as those for sshd and Apache, and is easily configured to read any log file of your choosing, for any error you wish.
      • Though Fail2Ban is able to reduce the rate of incorrect authentication attempts, it cannot eliminate the risk presented by weak authentication. Set up services to use only two factor, or public/private authentication mechanisms if you really want to protect services.
  • Tutorials
  • Block WordPress Scanners (not all fail2ban)
ConfigServer Security & Firewall (CSF + LFD)
  • Webmin
    • Has it's own Webmin module developed by ConfigSever.
    • ConfigServer Security & Firewall | Webmin
      • A Webmin module and an excellent CSF integration.
      • A stateful packet inspection (SPI) firewall, login/intrusion detection and security application for Linux servers.
  • Official Sites
  • General
    • CSF/LFD: Watches, among lots of other things, logs for login failures and blocks the offending IP via iptables. Also watches for modified system files, can detect port floods, use blacklists to block known hacker nets, limit connection count per source IP, and other stuff.
    • CSF is a comgbination of 2 programs, a firewall(CSF) and a login damon (LFD)
    • CSF fireall is an SPI firewall.
    • CSF is updated from within the fireall itself rather than the normal apt-get package route
    • The UI allows you to block/unblock IP addresses manually
    • CSF utilises mod security
    • LFD adds brute force detection
    • CSF features
      • DDoS preventions
      • Blocklist interation
      • GEOIP blocking / Country level blocking
    • WHM/cPanel uses CSF and LFD
    • ConfigServer Security & Firewall (CSF) | Virtual Architects Support Wiki
      • This is an excellent reference document on installation and usage.
      • LFD does more than just monitor log files for login failures.
      • LFD, in some opinions, is the best reason to implement the CSF firewall!
    • Mod_security and/or firewall for new setup - #5 by RJM_Web_Design - Virtualmin - Virtualmin Community
      • Yep. In CSF it’s called lfd, for login failure daemon. It’s pretty similar in function to fail2ban, which is why I don’t bother enabling both on the same server.
      • lfd can block individual IP’s or ranges based on user-determined criteria, for user-specified lengths of time. It can also convert persistent offenders from tempblock to permblock; execute external scripts to create block reports or unblock reports; notify the admin of failed and/or successful SSH logins, Webmin logins, and sudo elevations; and perform many, many other security functions.
      • All of those are in addition to the basic security provided by the main csf application, which includes functions like process tracking, system file integrity checking, mail volume monitoring, blocking based on public RBL’s, and a bazillion other user-configurable security functions. It’s a firewall, but it’s also much more.
      • I updated PHP 8.1 on six servers today (five production and one dev); and within a few minutes I received six emails and six text messages from CSF warning me that the files had changed. It also can inform root when users upload root-defined kinds of scripts, such as any script that sends mail; or when a user is sending out more mail in a given time period than some number specified by root.
      • It really is comprehensive.
      • It does take some time to learn if you want to maximize its usefulness. It also requires some version of syslog. I usually use rsyslog, but syslog-ng will also work.
    • What is CSF (ConfigServer Security and Firewall)? - ConfigServer Firewall, also known as CSF, is a firewall configuration script created to provide better security for your server while giving you an advanced, easy to use interface for managing firewall settings. CSF configures your server’s firewall to lock down public access to services and only allow certain connections, such as logging in to FTP, checking email, or loading websites.
  • Tutorials
    • Common CSF/LFD False Positives and How to Stop The Notifications - KnownHost - Learn more about common CSF/LFD false positives and a bunch more information that can help you manage your KnownHost server.
    • How to update Email Notification address for CSF/LFD – cPanel - Often at times, if you do not configure the email address for CSF/LFD notifications, it will cause server's EXIM queue to be filled up, as by default the notification will be sent to root, which the root user does not accept any local email deliveries.
    • Country Blocking / IP to Country Lookups / GeoIP / Geolocation
      • If you are running a network firewall such as pfSense, then do the Country Blocking in that device, so all network devices on your network can benefit from a single ruleset, but keep the lookup service enabled here to allow for IP to country lookups.
      • Do NOT use CC_ALLOW = ""
        • WARNING: CC_ALLOW allows access through all ports in the firewall. For this reason CC_ALLOW probably has very limited use and CC_ALLOW_FILTER is preferred
  • Installation
    • Before installing CSF, make sure you can login locally, either you can physical access the server and it's terminal or with a KVM  ypur provider supplies you that will be unaffected by CSF.
    • If you don't do this you can find yourself locked out permanently.
  • Upgrading
    • Easy upgrade between versions from within the control panel
    • Easy upgrade between versions from shell
  • Uninstalling
    • Follow these steps
      1. Login to PuTTy
      2. Copy tand paste the command below into PuTTy
        cd /etc/csf
sh uninstall.sh
      3. Press enter
  • Troubleshooting
UFW (Uncomplicated Firewall)
  • Virtualmin does not use this but webmin has a module for it.
  • Webmin UFW module --> UFW --> IPTables
  • UFW Essentials: Common Firewall Rules and Commands | DigitalOcean
    • UFW (uncomplicated firewall) is a firewall configuration tool that runs on top of iptables, included by default within Ubuntu distributions. It provides a streamlined interface for configuring common firewall use cases via the command line.
    • This cheat sheet-style guide provides a quick reference to common UFW use cases and commands, including examples of how to allow and block services by port, network interface, and source IP address.
  • UncomplicatedFirewall - Ubuntu Wiki - The Uncomplicated Firewall (ufw) is a frontend for iptables and is particularly well-suited for host-based firewalls. ufw provides a framework for managing netfilter, as well as a command-line interface for manipulating the firewall.
  • Linux Security - UFW Complete Guide (Uncomplicated Firewall) - YouTube
    • In this video series, we will be taking a look at how to set up, secure, and audit Linux servers. This video will explain and demonstrate how to set up and configure UFW and various firewall rules.
    • Skipped to 150s on purpose.

WAF

ModSecurity (ModSec / mod_security) (WAF)

ModSecurity is not currently installed by Virtualmin or officially supported.

  • Official Sites
    • OWASP ModSecurity | OWASP Foundation - ModSecurity is the standard open-source web application firewall (WAF) engine.
    • GitHub - owasp-modsecurity/ModSecurity: - ModSecurity is an open source, cross platform web application firewall (WAF) engine for Apache, IIS and Nginx. It has a robust event-based programming language which provides protection from a range of attacks against web applications and allows for HTTP traffic monitoring, logging and real-time analysis.
    • ModSecurity Frequently Asked Questions (FAQ) · owasp-modsecurity/ModSecurity Wiki · GitHub
      • ModSecurity™is an open source, free web application firewall (WAF). With over 70% of all attacks now carried out over the web application level, organizations need all the help they can get in making their systems secure. WAFs are deployed to establish an external security layer that increases security, detects and prevents attacks before they reach web applications. It provides protection from a range of attacks against web applications and allows for HTTP traffic monitoring and real-time analysis with little or no changes to existing infrastructure.
    • Reference Manual (v2.x) · owasp-modsecurity/ModSecurity Wiki · GitHub
  • General
    • Mod_security and/or firewall for new setup - Virtualmin - Virtualmin Community
      • mod_security is not related to network configuration. But, there is not mod_security GUI in Virtualmin.
      • Nothing stops you from enabling it, though. It’s a one-time thing; just install the package and turn it on (and configure it to use the rule sets you want to use, like the OWASP rules). mod_security is of marginal utility in a system that is well-maintained, but can be useful if you have old apps. mod_security is almost entirely a reactive solution; the rules are mostly based on past attacks, which have usually already been fixed in the software the attacks target. But, since most people aren’t very good at staying up to date, I’ve come around to thinking mod_security is pretty useful, and we’ll be adding it as a Pro feature in Virtualmin 8.
      • Yeah, I think it’s worth being clear that mod_security is what is often referred to as a “web application firewall”, but it does not operate at the same layer of the network stack as a traditional firewall. You should not group the two concepts together when discussing what you need to address a given threat model, and there is almost no overlap in what a network firewall can prevent vs what a web application firewall can prevent.
      • Web application firewalls (like mod_security) are also of limited utility in an up-to-date well-maintained system.
    • Web-based Application Firewall (WAF) - Rocky Linux Documentation - mod_security is an open source web-based application firewall (WAF). It is just one possible piece of a hardened Apache web server setup. Use it with, or without, other tools.
    • Mod_security and/or firewall for new setup - #3 by RJM_Web_Design - Virtualmin - Virtualmin Community
      • Yeah, I think it’s worth being clear that mod_security is what is often referred to as a “web application firewall”, but it does not operate at the same layer of the network stack as a traditional firewall. You should not group the two concepts together when discussing what you need to address a given threat model, and there is almost no overlap in what a network firewall can prevent vs what a web application firewall can prevent.
      • Firewalls are one thing, and Virtualmin sets up a basic Firewalld firewall for you and provides a GUI for it, so you can add whatever additional rules you like. That is wholly orthogonal to mod_security, which is a layer 7 rule-based tool for preventing some kinds of web application attacks.
      • Firewalls (like Firewalld or CSF) are of limited utility in a server environment. Tools that actively monitor behavior and add rules (like Fail2ban, which Virtualmin includes and has a GUI for, or sshguard or I think CSF has its own similar thing) based on things that look like attacks are very useful, though.
      • Web application firewalls (like mod_security) are also of limited utility in an up-to-date well-maintained system.
  • Tutorials
  • Diagnostics
Comodo WAF (mod_security with Comodo rule set)
cPGuard
Naxsi WAF

Malware Scanners

ClamAV
  • Be able to scan home directories with ClamAV (clamscan)
  • Webmin Module
  • General
    • Is it safe to update ClamAV manually? - General Discussion - Virtualmin Community
      • In my experience, things get messy. ClamAV packaging has been a wild west situation for almost its entire existence, with the ClamAV folks providing guidance for packagers that is unreasonable and example configs that don’t work, and every packager that comes along tries to make sense of it and ends up solving the problems differently and in incompatible ways (though the Debian/Ubuntu packages have been somewhat more stable than the EPEL/Fedora packages, which have had at least three or four incompatible variations, which were also incompatible with a couple of third party packagers).
      • If I were you, and if I really felt compelled to upgrade, I would test on a development server before trying it on a production system, if you really depend on ClamAV working reliably, because I’d bet on breakage.
    • Regarding Antivirus - #5 by Stegan - Virtualmin - Virtualmin Community
      • Our default AV in a Virtualmin installation is ClamAV, and we provide GUI support for it. It is quite resource-intensive, but it does work reliably and is reasonably effective at detecting viruses and malware.
      • Antivirus (any antivirus, not just ClamAV) generally can’t protect servers from most malicious attacks. The kinds of malicious attacks that servers face are rarely mitigated by antivirus software. That’s just not the vector by which servers are usually compromised.
      • Antivirus is among the least effective ways to spend your time when trying to secure a server. I won’t say it is completely useless, but it’s quite far down the list of things to do to secure a server.
      • But, you can certainly run any antivirus you want on your server. It’s your server. Virtualmin is not an operating system, it is a management tool, it is not preventing you from doing things on your server as though it were a normal RHEL/Alma/Rocky/Ubuntu/Debian server, because it is a normal installation of your OS with our software installed on it. We use standard OS packages wherever possible.
      • Add 4.000.000 signatures to Clamav antivirus - Protect your computer against 0-day malwares with ClamAV! Discover how to increase the detection of your antivirus now
Maldet
Immunify360

chroot / chroot jail / Jailkit

chroot = Change root directory

  • General
    • Not another chroot Question? chroot explained? - Help! (Home for newbies) - Virtualmin Community
      • This is my forum thread.
      • Need confirmation of these
        • chroot = Change root
        • Aesthetic only
        • Chroot only works on
          • port 22 for both SFTP and SSH
          • and the Terminal in the users Webmin
        • ProFTPd controls SFTP on port 2222 and therefore is unaffected by the Virtualmin implementation of chroot.
        • You configure restrictions in ProFTPd.
        • You control what functions and services are added into the Jail by using the jail manager
        • It restricts what commands can be run in SSH for the user. You can add what is allowed in.
        • Any functions/services to be used in the jailed session need to be added.
        • It is not a security feature, but only ‘security via obscurity’
        • Jails are not very useful, it’s just a thing people in the hosting world like. Hides a load of mess from their clients.
        • Chroot does more than jails.
        • The Proper name for this, in the way we are using this feature = chroot jails.
        • If you are not giving your clients SSH access, chroot is pointless.
        • chroot needs root to run and is why it can be dangerous.
      • Questions
        • Why aesthetic only if you can restrict what functions a user has access to with SSH?
        • Where do you configure the SFTP (port 2222) restrictions in ProFTPd?
          • Is this done by hand
          • Webmin → servers → ProFTPD Server
          • ProFTPd jail features?
          • FTP is already restricted to the home page.
        • Does this stop people FTP’ing to the root and seeing files?
        • Does this stop people using SSH getting to the root of the server?
    • My clients access the virtualmin shell automatically as root - Virtualmin - Virtualmin Community
      • Joe: Webmin modules are root access tools, by default. Some can be locked down, but in this case, Virtualmin already has support for granting users Terminal access. You should not grant them access to the Terminal Webmin module. They don’t need it.
      • Tooltip: Be very careful with this option, as most Webmin modules default to providing dangerously complete control over the services they manage.
    • Virtual Server vs. Chrooted Virtual Server - Virtualmin - Virtualmin Community
      • Joe
        • Q: What is the security benefit of chrooted virtual servers vs. normal virtual servers?
        • A: chroot is not a security feature, despite the widespread belief that it is. It basically just hides some filesystem details from the logged in user. And, in fact, a chroot jail can open serious security holes if you don’t understand the implications of putting things into the jail. Though, most of the security risks of chroot jails have been resolved by use of capabilities in the Jailkit packages we provide, I am not entirely confident there aren’t still ways to shoot yourself in the foot. chroot has such a long history of exploitable usage that I am hesitant to say anything nice about it (we added it only after capabilities became universally available across all supported distros and in Jailkit).
      • Ilia
        • Jailkit doesn’t give you any extra security in this regard, and security via obscurity is not really considered. Besides, Jailkit is an additional complexity. There is no need in enabling it to make your server more secure. For instance, I am not using it.
        • If an app (e.g. like WordPress plugin) that runs on one virtual server is hacked, the attacker will not be able to access files under different virtual servers as they are owned by a different user, as PHP scripts are executed as given virtual server owner, although, the hacker would most probably be able to get a list of /home directory (that would depend on PHP configuration though), i.e. see the names of other virtual servers hosted, so if you want to prevent this from happening you would have to use chroot setup.
    • FTP and SFTP for ProFTPD - Virtualmin - Virtualmin Community
      • Joe
        • ProFTPd offers SFTP on 2222 (and FTPS on the usual ports). OpenSSH runs on 22 and also happens to offers to offer SFTP, but its primary purpose is for ssh access…the two can’t share the port, so ProFTPd goes on 2222.
        • If you need the controls that ProFTPd provides (like not allowing running programs), you need to direct users to 2222 or FTPS and don’t give them an SSH login account.
    • Chroot in virtualmin - #3 by gerhard - Developers - Virtualmin Community
      • Joe
        • This is a really old thread, don’t make zombies! None of it is relevant anymore, as chroot jails for both ssh and FTP over SSH is supported right out of the box…you don’t need to configure SSH jails, you just need to turn them on (this uses Jailkit, configurable in the UI, though most folks don’t need to do much with the configuration…some folks may need to add other binaries to the jail).
        • And, FTP over SSH is always available on port 2222 (this use ProFTPd jail features and doesn’t need configuration).
    • Questions about chroot and Virtualmin. | Virtualmin
      • joe
        • I would argue you shouldn't configure chroot ever, if you're using it for security. There are some pretty significant dangers to using it as a security tool. For one, it breaks some of the security features of ssh. For another it introduces a stage in the interaction with your user where they have root privileges (chrooting requires root privileges). If you make a mistake, or there is any insecure element in your chroot configuration, and an exploit occurs it could be dramatically more dangerous than someone merely seeing a few files in /etc. So, while it makes the system seem more secure at first glance, it actually probably makes it dramatically more likely to be rooted.
        • In short, we don't recommend chroot environments. If you need root-like levels of separation, there <i>are</i> good methods for achieving it (Xen, Zones, vservers, etc.), and we have tools for managing those methods (we have a new product in private beta now and entering public beta this week for managing virtualized systems).
    • Virtualmin + SFTP + chroot – The Research Lab
      • This guide examines setting up chroot’ed SFTP-only user accounts under Virtualmin.
      • SFTP is a secure alternative to FTP and FTPS that uses SSH.  With this setup, no FTP server is needed, as the native sshd server is used instead, SSH does not require an SSL certificate (like FTPS), and is usually considered more secure.
      • However, one drawback is that FTP servers typically offer a simple config option to “restrict access to the user’s home directory”, whereas SFTP requires a chroot’ed setup to do this, which is more complex, and not supported natively by Virtualmin (or really any other CP).
    • Virtual Server vs. Chrooted Virtual Server - #4 by dragonsway - Virtualmin - Virtualmin Community
      • Q: Or is the only way to truly achieve that level of security is by chrooting the Virtual Server?
      • A: Jailkit doesn’t give you any extra security in this regard, and security via obscurity is not really considered. Besides, Jailkit is an additional complexity. There is no need in enabling it to make your server more secure. For instance, I am not using it.
      • Q: How does using a normal Virtual Server, per user, prevent a malicious actor from hacking a random user’s Virtual Server and somehow gain access to the core server itself?
      • A:
        • Basic
          • At first, simply make sure that you use standard Virtualmin installation, i.e. install it on a clean state instance, using official install.sh script and that installation goes well (all installation steps are completed successfully).
          • Later, make sure that you use strong passwords for your virtual servers, as well as strong passwords for SSH/Webmin/Virtualmin/Usermin logins.
          • Try using key authentication for SSH and disable plain password authentication (at least for root user).
          • Also, enable two-factor authentication for Webmin/VIrtualmin/Usermin logins.
          • Login failure daemons, like fail2ban will also make it more difficult to brute force your passwords. This is mainly it.
        • Now, in terms of inter-user security
          • if you want to isolate users from one another, simply always create a parent virtual server, which will setup a separate Unix user, as separate Unix user is the main layer of security that just works naturally.
          • If an app (e.g. like WordPress plugin) that runs on one virtual server is hacked, the attacker will not be able to access files under different virtual servers as they are owned by a different user, as PHP scripts are executed as given virtual server owner, although, the hacker would most probably be able to get a list of /home directory (that would depend on PHP configuration though), i.e. see the names of other virtual servers hosted, so if you want to prevent this from happening you would have to use chroot setup.
        • Default Virtualmin installation is very secure by default.
          • Try not to deviate from configuration of what standard installation provides, unless you know exactly what you’re doing. And remember, extra complexity almost always highers the risks of bringing in more potential issues to the field.
      • SSH/Terminal enabled for concern - Virtualmin - Virtualmin Community
        • When I try to enable the terminal and SSH for a website, I notice that each domain owner has read access to the root directories. Is there a way to restrict read access of all files outside the virtual domain directory?
        • You can use jailkit, but to use jailkit you must be aware that all resources that the user needs must be placed in the jail, you also may have to alter the users environment to get the best experience. That said for what reason does a domain owner ssh access ? As most things that a domain owner may need are in the virtualmin/usermin panels. I don’t give Domain owners ssh access, I point them to the relevant function in Vmin/Umin.
        • "be aware that all resources that the user needs must be placed in the jail"
          • Note that Virtualmin Pro users with version 7.9.0 and up, will be able to do it with ease.
        • We setup by default the following sections: 
          perl, basicshell, extendedshell, ssh, scp, sftp, editors, netutils, logbasics
        • "You said php worked out of the box. It does not appear to do that"
          • It does on EL systems. Debian and derivatives don’t have php section defined, so it has to be added manually.
          • Virtualmin Pro can still copy php binary and all dependencies using Extra commands and directories for Jailkit to copy option.
        • "So i have to guess what dependences are required for php?"
          • Nope, this is what Jailkit init program is doing.
          • I think the more we add UI enhancements that make it seem like an easy thing to do, the more likely someone will make mistakes that make their jails breakable (which is actually extremely dangerous on Debian/Ubuntu where the Jailkit binaries do not use capabilities and are running as full root, at least, last time I checked…the RPM uses capabilities, so it’s much safer).
          • The distinction lies in the packaging differences between EL and Debian distributions. Specifically, in the EL distribution, jk_init includes a defined [php] section, whereas this section is absent in the Debian distribution.
          • However, this difference should not affect our users, thanks to a new feature in Virtualmin Pro that automates the process. Users simply need to add php to the Extra commands and directories for Jailkit to copy field, and the system will handle the rest seamlessly.
        • I was always assuming the RPM was packaging the upstream files unmodified, but I guess not. I see there is no php section here: [jailkit] Contents of /jailkit/ini/jk_init.ini
          • I wonder now what is adding [php] section in RPM package?
          • Most probably Red Hat or EPEL?
          • It is an EPEL package, so, whoever maintains the Fedora package.
        • But, also, I don’t know how we make it clear that folks need to understand Jailkit in order to use it safely!
  • What is chroot?
    • chroot - Wikipedia
    • Jailkit - chroot jail utilities
      • Jailkit - a set of utilities to create chroot shells or chroot daemons
      • Jailkit is a set of utilities to enhance the possibilities of chroot jails. Jailkit contains a set of tools and config files to automate the deployment of chroot jails. Jailkit also contains various tools to limit user accounts to specific files or specific commands, configured from a config file. Setting up a chroot shell, a shell limited to some specific command, or a daemon inside a chroot jail is a lot easier and can be automated using these utilities.
    • Jailkit - chroot jail utilities (jailkit 8)
      • Jailkit - a set of utilities to create chroot shells or chroot daemons
      • Jailkit is a set of utilities that can limit user accounts to a specific directory tree and to specific commands. Setting up a jail is much easier using the jailkit utilities that doing so 'by hand'. A jail is a directory tree that you create within your file system; the user cannot see any directories or files that are outside the jail directory. The user is jailed in that directory and it subdirectories. The chroot(2) system call is used by jailkit to put the user inside the jail..
      • If you want the user to be able to do just one thing, you can set up the jail so that the user is able to do exactly and only that one thing. For example, if you want the user to be able to run scp, you install a copy of scp in the jail along with just enough support to execute it (e.g., using a limited shell). As you can understand, the fewer executables you have in a jail (and the more their capabilities are limited such as using strict configurations), the more work a hacker needs to break out of it. It is important to note that a chroot jail can be easily escaped if the user is able to elevate to the root level, so it's very important to prevent the user from doing so..
      • A badly configured jail is a security risk!
      • If a jailed user or a jailed process can modify files in (for example) the JAIL/lib/ or JAIL/etc/ directory (i.e., those within the jail directory), the user can bypass security checks and gain root privileges.
      • No directory inside the jail except for the user's home directory or tmp should be writable by the user. Especially the root of the jail should not be writable by the user.
    • Jail Management » Linux Magazine
      • This is a well written article explaining chroot and jails.
      • Setting up chroot jails is no simple task. Jailkit can make this job a little easier by automating setup and configuration.
      • chroot is a way to limit a user account's access to the parts of the directory tree by – as the name of the command implies – changing its root directory. The result is what is known as a chroot or, sometimes, a chroot jail, which draws on the larger system's resources as needed
      • Contrary to widespread misinformation, a chroot is not a security measure unless specifically configured as one.
      • Although confinement in a jail can limit what an uninformed user can do, expert users could escape a jail by creating a second jail within the first.
      • In addition, any process run with root privileges can access resources outside the chroot.
      • Similarly, if a user has permissions for any files outside their home directory, they are not jailed.
      • In addition, any user with root privileges can access the chroot from the main system, including those using sudo
  • Jail configuration
    • Config file: 
      /etc/jalkit/jk_init.ini
    • Webmin --> System --> Jailkit Jail Manager
      • Tooltip:
        • Jailkit is a set of utilities to limit user accounts to specific files using chroot() and or specific commands. Setting up a chroot shell, a shell limited to some specific command, or a daemon inside a chroot jail is a lot easier and can be automated using these utilities.
        • This module provides a user interface for managing the Jailkit jail configuration file (jk_init.ini). With it, you can create, modify, and delete jail definitions.
  • Enable disable chroot for a User/Virtual Server
    • Virtualmin --> System Settings --> Server Templates --> 'A Server Template' --> Administration user --> Chroot jail new domain Unix users
      • Tooltip: This option determines if new top-level virtual servers are by default setup to chroot the domain owner Unix user into a directory that is isolated from the rest of the system.
    • Virtualmin --> Virtual Server --> Manage Virtual Server --> Edit Owner Limits --> Other restrictions --> Chroot jail domain Unix user
      • Tooltip: If enabled, all SSH and SCP access by the virtual server's Unix user will be restricted to his home directory. This also applies to PHP scripts run in FPM or FCGId modes.
  • How to use chroot in virtualmin
    • debian - How to use Jailkit Jail Manager in Virtualmin to restrict users - Unix & Linux Stack Exchange
      • Q:
        • How to use Jailkit Jail Manager in Virtualmin (Webmin 1.892) to restrict users in their homes including virtual website and all services running under user?
        • I am setting up small website hosting service and I must disable access to everything except iser's home.
        • I don't want to use FTP or FTPS! User's will have full SSH access to theri system and they will be able to run for example NodeJS scripts, Teamspeak, etc...
      • A:
        • Virtualmin --> 'Virtual Server' --> Manage Virtual Server --> Edit Owner Limits --> Other restrictions --> Chroot jail domain Unix user: Yes
        • Setting this in a 'Server Template' for your client's first is better.
    • CHROOT issues/questions - Help! (Home for newbies) - Virtualmin Community
      • jimr1
        • I would only use a chroot on a user that has, and will use shell access, and also use the correct utilities to add whatever the sys admin deems fit.
        • As Joe pointed out chroot has not much effect to a web user as they have limited access to the system.
      • Joe
        • You need to add anything you want your chrooted user to be able to use to the jail. To use sendmail, you need to add that command and its libraries to the jail, either using the jk_cp command or adding it to the config file for the jail being created for your users (there are several included jail configs, we default to a quite limited one, for security reasons).
        • This is true of any use of a jail, regardless of control panel being used to manage things. A chrooted user only sees what you put in their chroot. If you want them to use PHP, you gotta give them php (and maybe the specific extra versions they need, if any). If you want them to send mail, you’ve gotta give them sendmail. As I said, there are some defaults included with Jailkit, or you can make your own and tell Virtualmin to use it.
        • The point of a chroot is to restrict what the user can and can’t see and do. It’s quite restrictive by default (and by necessity…it’s not all that hard to accidentally give users the ability to escape the chroot).
      • jimr1
        • a domain owner does not really need access via ssh really with there priv level they can do nothing to the system barr look at directories they have privs to … This is the point of webmin/virtualmin it negates the need for ssh access as webmin/virtualmin has:
          • a File manager that can cover most file operations (upload/download/new/delete + more)
          • a terminal that gives the same access as ssh
          • a lot of things a domain owner would need wrapped up in the virtualmin gui
        • so what do you think the user would benefit from having raw ssh access ?
          • As the virtualmin terminal is that good, to the point I have not broken it yet, On my own server I am thinking of removing native ssh access and using just the virtualmin terminal if the user ever wants to use it.
          • I have found most domain owners seem to use the file manager and the virtualmin menuing system to edit what they need and very seldom use the terminal.
        • It may be true other panels may ‘nurse maid’ a jail by adding virtually everything to the jailed user but perhaps that is not required and the sys admin (i.e you) should have a total say what system files are added to the jail to avoid possible break outs of the jail. Maybe this is just a way of an new way of sys admin to you but it does work, but I guess each to their own
      • Joe
        • The distro had nothing to do with defining the jail configurations (though I guess they could, jails are not a thing most people care about). The upstream jailkit source provides them (and as far as I know, they’re mostly unchanged by the distros that do package jailkit, and our package for RPM-based distros does not alter the jails…we used to fix a bug in one of the jails from upstream, but it’s now been fixed upstream, and we no longer customize it).
        • There isn’t any judgment happening at the distros. They don’t care. It’s just another package to them, and it’s a very rarely used package in most contexts; you won’t find any Debian/Ubuntu/RHEL core documentation about Jailkit, because jails are not very useful, it’s just a thing people in the hosting world like.
        • But, the idea is that you’ll configure the jail to suit your needs or the needs of your users, and with the commands you’re comfortable with them having. There are a handful of predefined jail configurations, and you can create as many of your own as you want. I guess we should spend more time on either documenting that or making the default jail do the usual things people expect to be able to do when they ssh in (but that negates most of the already small security benefit of a chroot jail).
      • Joe
        • We use Jailkit for Jail creation and management (probably most others do, too, except maybe cPanel who have a lot of their own in-house tools and custom build everything), so the Jailkit site is a good place to start: Jailkit - chroot jail utilities
        • Last I checked Debian (and Ubuntu) did not build Jailkit with capabilities, and so they are more likely to be dangerous than on RPM-based distros, where we provide the package and it has capabilities enabled.
        • The chroot is created with full root privileges on those distros, and if exploited at that stage, it would potentially provide root access to the system, not merely a chroot escape (having it build to use capabilities means it only has the ability to create a chroot and maybe one other privilege I can’t recall, so it’s less of a threat, though still potentially problematic).
        • So…I kinda think using chroot jails on those distros is negative for security. The likelihood of an exploit is probably pretty small, if you are careful about what you put in your jail(s) and what permissions they have. It’s an old codebase, and has had lots of time to become well-understood. I recommend reading and understanding this specific page, in particular, Jailkit - chroot jail utilities before using jails.
      • Joe
        • To send mail using the sendmail command, you need to add the sendmail command to the jail, either via jk_cp (for one user jail) or by adding it to the jail configuration file (which will add it to future created jails).
      • Joe
        • And, you will find the default jail configurations in /etc/jalkit/jk_init.ini, and you can modify those, and you can choose which kind of jail is used by Virtualmin (that’s chosen in Server Templates, I believe).
      • Stegan
        • I still don’t understand what the motivation is to use jails?
        • They appear to add nothing but trouble. based on some alleged benefit of additional security.
      • ID10T
        • Worth a read is the Wikipedia entry.
          • A chroot on Unix and Unix-like operating systems is an operation that changes the apparent root directory for the current running process and its children. A program that is run in such a modified environment cannot name (and therefore normally cannot access) files outside the designated directory tree. The term "chroot" may refer to the .mw-parser-output .monospaced{font-family:monospace,monospace}chroot(2) system call or the chroot(8) wrapper program. The modified environment is called a chroot jail.
        • Also from Linus Torvald 
          So all chroot(2) really does is reset the “/” reference?
          • Yes. Literally. Everything else stays the same, including any open files (and cwd).
          • It’s a “flaw” in chroot if you consider it a jail, but it’s used for so much more than that.
        • Note that the most common use of chroot isn’t actually the “jail” kind of usage, but building and installation environments (ie a lot of package building stuff end up using chroot as a way to create the “target environment”).
          • chroot safety - DEV Community - As seen before, chroot isolates the 'outer' filesystem from a new process started with the command. It's handy, but not safe. With some creativity the process can break its 'chroot jail'.
      • Joe
        • chroot has loads of great uses. A jail is the least interesting, but in web hosting, it became the norm for aesthetic reasons (people didn’t like their customers seeing the rest of the system), so everybody expects us to offer it, so we do.
      • Joe
        • It is mostly aesthetic. What it looks like to a non-technical user to see a list of other user homes when they ls /home. That looks scary, and may bother hosts who don’t want their customer lists being visible (which is reasonable). So, requiring use of ProFTPd connections (whether FTPS or SFTP) can prevent that without needing a chroot jail, if those customers don’t need a shell.
  • Results of my testing
    • When you swap between chroot on/off, the change is not immediate, wait 1-2 min.
    • chroot option only affects SSH on port 22.
    • ProFTPd controls SFTP on port 2222 and therefore you configure restrictions in ProFTPd.
    • With chroot off
      • FTP (port 21)
        • I can only see my test user's home directory.
      • SFTP (port 22)
        • home directory = /home/testuser/
        • I can go above my test user's home directory.
        • I can see other home directories, but cannot access them.
        • I can see server root (/).
        • I can see at lots of files and folders in the root.
        • = Not Restricted by chroot
      • SFTP (port 2222)
        • home directory = /home/testuser/
        • I can go above my test user's home directory.
        • I can see other home directories, but cannot access them.
        • I can see server root (/).
        • I can see at lots of files and folders in the root.
        • = Not Restricted by chroot
    • With chroot on
      • FTP (port 21)
        • I can only see my test user's home directory.
        • = Restricted by ProFTPd
      • SFTP (port 22)
        • home directory = /home/testuser/
        • I can go above my test user's home directory.
        • I cannot see other home directories.
        • I can see server root (/).
        • I can see some files and folders in the root, but not all of them.
        • = Restricted by chroot
      • SFTP (port 2222)
        • home directory = /home/chroot/12345612345699/home/testuser/
        • I can go above my test user's home directory.
        • I can see other home directories, but cannot access them.
        • I can see server root (/).
        • I can see at lots of files and folders in the root.
        • = Not Restricted by chroot

Housekeeping

  • Is there a reason that Virtualmin keeps so many previous kernals? - Help! (Home for newbies) - Virtualmin Community
    • Virtualmin does no such thing. Virtualmin is not your package manager.
    • Just run the following after kernel updates: 
      apt clean && apt autoclean && apt autoremove
      • You can also automate this by creating a bash script that runs the above whenever a kernel update happens.
    • Q: Yes… but we are always told to not do things outside of Virtualmin… So if we are updating via Virtualmin…
    • A: Though I would recommend being careful with autoremove. You need to read what it’s doing and make sure you understand what it’s removing before approving it.
    • Note the search term: linux-image
  • Update Detected Operating System
    • Update Detected Operating System - What now? - Virtualmin - Virtualmin Community
      • Joe
        • It literally updates the version number in the dashboard. I agree it feels weird to make it a dramatic looking thing. Edit: Though, I wonder if a major version bump would lead to new config files (the files that determine defaults for the OS) being copied in some cases. I’ll have to check.
        • By the time the notice appears there is nothing to install. The OS upgrade is done (and Webmin didn’t do it…Webmin never automatically updates anything, you would have had to have approved any upgrades Webmin did…you might also have automatic updates enabled at the OS level, but that’s none of Webmin’s business). But, minor version updates are just that. They aren’t something to get crazy about. You should have regular known-good backups, of course, but minor updates are not expected to be disruptive.
      • Ilia
        • By clicking this button, you update the Webmin configuration to match the current minor version of the OS. If Webmin has new settings for the updated OS version, these will be added to the config (on the next Webmin upgrade). Changing the OS completely or distro upgrading might cause issues, though it depends.
        • However, there’s no need to worry in your situation. The Ubuntu updates you’re dealing with are minor and don’t have any major changes that could cause problems. You can click on the link to read the Release Notes for more details about the minor release. Or, just click the ‘submit’ button to update the Webmin config and remove the alert.

Troubleshooting

  • Diagnostics
    • check-config (CLI) – Virtualmin 
      virtualmin check-config
      • This program checks your system's Virtualmin configuration, outputting the progress of the check as it goes. If any serious problems are found it will halt and display the error found.
      • This program can automatically update some configuration files if needed (i.e. if Apache is configured to use a PHP version that's not installed).
    • Troubleshooting Websites | Virtualmin — Open Source Web Hosting Control Panel
      • Web server configuration - Troubleshooting web server issues involves checking various elements, from configuration settings to log files. Common problems are not always evident as errors in error_log, so a comprehensive approach is needed.
      • Webserver logs - The first step in troubleshooting is to examine the log files. Each virtual server or sub-server in Virtualmin has its own log files located in /home/example/logs (replace example with your server name). The error_log is typically the most informative, but access_log may also provide useful insights.
  • Using the logs
    • Email from Client doesn't always work - Virtualmin - Virtualmin Community
      • Modern systems send most logs to the journal. You should get familiar with it (the journalctl command is the standard tool for searching/tailing logs in the journal). The postfix, dovecot, and saslauthd units are probably the relevant ones for your problem.
      • Webmin has the System Logs Viewer module that defaults to include the journal (instead of the System Logs module, which works with various syslog implementations). But, for anything complicated, the journalctl command is still your most capable option.
    • `System Logs` missing on Ubuntu
  • Webmin GUI not working as expected after an update.
    • Theme Configuration --> Clear Cache
  • Forced Refresh system information
    • Click the refresh button at the top right of the dashboard, it will perform a "Force system information refresh"
    • This is useful if modules are missing on the dashboard
  • Locked out of Webmin/Virtualmin / Your IP has been blocked
  • Connection issues
    Firstly, make sure you are not locked out of your system by the firewall (i.e.your IP is banned), it might appear as a connection issue.
    • Check the following settings are correct
      • Webmin --> Networking --> Network Configuration --> Hostname and DNS Client --> DNS servers:
        • This has different options depending on how you set DNS resolution in the systemd-resolved (DNS Resolver) but htere must be a valid DNS resolver that can be accessed defined here.
          • 10.0.0.1
          • 127.0.0.53, then 10.0.0.1
          • 9.9.9.9 or 8.8.8.8 etc.. (if not DNS hijacking and/or just using external DNS)
      • Webmin --> Networking --> Network Configuration --> Routing and Gateways --> Default router: this is set to 'None (or from DHCP)', change this to 'Gateway: 10.0.0.1'
    • If you can access you server locally but not from the outside check the following
      • NIC
        • Check to see if the gateway is set on the Virtualmin server's NIC
        • Check which route is being used if you have more than one NIC
      • DNS
        • Split DNS configuration in your router - If this is being used do you have the correct entries.
        • Are the DNS and nameserver entries correct at your registra
        • Host file entries on your local computer
        • Are the router DNS override entries correct.
        • Do you have DNS hijacking running on the router and this is causing issues.
        • Are your virtual servers configured to use your external IP address.
      • Routing
        • Is your router (ie LuCI/LineageOS or pfSense) running a Webserver on port 80 and 443, if so, change these ports on the router.
        • Have you setup port forwarding / NAT properly
        • NAT Reflection (optional) - If enabled, is this configured and running correctly
        • Is there a firewall blocking ports 80 and 443 on the router
        • Remove IPv6 from the router. Not everything supports this correctly.
      • Virtualmin
        • Virtualmin --> System Settings --> Re-Check Configuration
          • this makes sure there are no obvious issues
        • Create a new Virtual server with a random domain name:
          •  i.e. chocolatefactory123.com
          • use windows host file override an see if it loads normally.
          • This might also fix the other sites.
          • don’t install a lets encrypt SSL
          • When I did not have any configure virtual servers adding one finished whatever Virtualmin needed to do and then it works so it might help here.
        • You have to look at the logs.
        • Put the website on the correct IP address
          • Virtualmin --> Manage Virtual Server --> Change IP Address
        • Virtualmin --> system Settings --> Re-Run Install Wizard
          • This is useful if you have made some changes and things are not working.
          • Do this last to prevent issues and potetially wiping out some of your settings.
          • Update Incorrect IP Addresses if prompted.
        • Virtulamin --> Virtualmin Configuration --> Configuration category: Networking settings --> Network interface for virtual addresses
          • Check this is configured correctly, especially if you have 2 network cards.
          • Tooltip: If your server has multiple interfaces, you may choose which interface to use for your virtual domains. If SSL or FTP virtual hosts are enabled, a new IP will be required for each domain on which the feature is enabled. Unless configured otherwise during domain creation, the new addresses will be created on the interface specified here.
    • Related links and articles
      • DNS Frequently Asked Questions – Virtualmin
        • Virtualmin error: 127.0.0.1 isn't listed in /etc/resolv.conf
        • How do I setup nameservers for my server?
      • Https website unreachable - Help! (Home for newbies) - Virtualmin Community
        • Take luci/lineageos gui off port 80 http and 443 Https
        • From what you have told me it might not be a routing issue but a misconfiguration some where.
        • Create a new Virtual server:
          • with a random domain name, i.e. chocolatefactory123.com , use windows host file override an see if it loads normally. This might also fix the other sites.
          • don’t install a lets encrypt SSL
          • When I did not have any configure virtual servers adding one finished whatever Virtualmin needed to do and then it works so it might help here.
        • The ip in the example.nl virtual hosts file 10.xx.xx.10 change to 178.xx.xx.27 and remove the ipv6 address (for now) and restart apache, then try
          • Virtualmin --> Manage Virtual Server --> Change IP Address
        • The issue - 2 NICs
          • Ok, you have a internal and external address, weird.
          • Ah, it’s start to ring a bell. Not, not weird. This is a rackserver with 2 connections. It needs both since it acts as a node, where the other network is used for internal communictaions between nodes. I think that’s where it might confuse this setup.
          • I feel I should try and rerun the wizard and try to force him to ignore the 10.xx.xx.xx addresses. Until now I thought this was somehting that virtualmin created himself to handle internal requests…
          • Not sure how you would make eth1 the main IP. I can’t remember selecting the IP in the wizard.
          • It will allow me to change the ens3 to ens4 and come up with this refresh option for all domains. It worked instantly.
          • Ran `Re-Run Install Wizard`
          • Updated Incorrect IP Addresses.
  • Resetting back to initial values as set in your Server Template
    • Virtualmin --> Limits and Validation --> Validate Virtual Servers --> Reset Features
    • This feature reset values of your Virtual Server back to how they are specified in the asigned Server Template.
    • You can select multiple sections to reset.
    • Caution is advised as I do not know if this will delete databases and email accounts etc... if in doubt make a backup first.
  • No 7z file support in File Manager
    • You get the following error in the File Manager
    • How to Use 7Zip in Ubuntu and Other Linux
      • Cannot extract .7z file in Linux? Learn how to install and use 7zip in Ubuntu and other Linux distributions.
      • the 7Zip package in Linux is named p7zip, starting with the letter ‘p’ instead of the expected number ‘7’.
    • Run the following command from the terminal 
      apt-get install p7zip-full
  • Folders extracted from 7-zip (7z) archives in File Manager are 700
  • Dashboard Web Terminal does not work and comes up with the following error 
    Failed loading terminal : WebSocket connection error

  • Show password button is missing
  • Dashboard is showing the wrong IP address for the 'System hostname'
    • The IP address being shown is an old DHCP IP address that this server used a long time ago while being setup.
    • Virtualmin --> Re-Check Configuration
    • Webmin --> Networking --> Network Configuration --> Network Interfaces
      • Check the Static IP is correct.
      • Check you are not using DHCP instead of a static IP
    • Webmin --> Networking --> Network Configuration --> Host Addresses
      • If you see the IP address listed here, edit it and change it to the correct IP address.
      • Restart the server or you might be able just to flush/refresh DNS.
  • Right clicking on the Virtualmin tab no longer opens the dashboard
    • This is not a feature of Virtualmin.
    • Theme Configuration --> Default page for Virtualmin
      • This is only when you open Virtualmin for the first time (i.e. login.)
    • Right clicking on the Virtualmin tab opens the same page in another tab only, so if you are on the dashboard it will open a new tab on the dashboard.
    • If you are on the Webmin tab, when you click on the Virtualmin tab then the first vitual server will be opened on the 'Virtualmin Virtual Servers' page.
    • Workaround
      • Right click on the Virtualmin tab to open a new tab
      • Left click on the Virtualmin tab to take you to the dashboard, or just click on the dashboard link or icon in the menu.
    • On Webmin tab, Virtualmin tab right click does not respect `Default page for Virtualmin` · Issue #796 · virtualmin/virtualmin-gpl · GitHub
      • This fixes the issue on newer versions of Virtualmin.
  • Dashboard - Display Corruption
    • Webmin/virtualmin display corruption (term, server graphs) - General Discussion - Virtualmin Community
      • The server usage graphs, terminal module and favicon are corrupted - they are filled with vertical colored lines
      • I managed to get the terminal module fixed by disabling the webgl extension
      • This is nothing to do with the server or Virtualmin but is caused by your browser. I had this issue and discovered it happens when the canvas is blocked. If you are using an extension like Canvas Blocker or LibreWolf browser you can whitelist the domain. Or you can disable the “Enable ResistFingerprinting” setting in LibreWolf. It also seems to work fine in other browsers.
      • For Firefox one can add/edit the additional permission for a given domain and allow the site to “Extract canvas data”
  • Cannot switch php execution modes
    • Cannot switch php execution modes - Help! (Home for newbies) - Virtualmin Community
      • Why do you want to switch to FCGI?
        • Fcgi allows you to have a different version of php running on different directories
          Example ~/public_html executes with php8.1 and ~/public_html/oldstuff executes with php7.4 this is configurable via virtualmin
      • Did you migrate from another system to the new one?
        • Either way, on Ubuntu 22.04 you can fix it by setting: Virtualmin --> Web Configuration --> Website Options --> CGI script execution: suEXEC

Developers Only

 

 

 

Installation Instructions

Follow the instructions below in order. and do not create any accounts until your are told to.

Not every setting is mention, but I have outlined the mains one to get you going and have a good setup to work with.

Preperation

  • Install your choosen Linux server / Base OS
  • Choose a hostname for your server (eg: server.example.com)
    • This is the name that you will call your Virtualmin server and it needs to be a Fully Qualified Domain Name (FQDN)
    • You can't use this hostname (eg: server.example.com) for a virtual server as it will break things in particular the routing/mapping of the email service.
      • Best practices for choosing the system hostname during setup - #4 by hennie.dv - Webmin - Virtualmin Community
        • “If your system does not have a fully qualified hostname, the script will ask you to provide one. The name of the system can be anything you want, but it must be fully qualified and should not match a name you’ll be hosting mail for. For example, if you have domain virtualmin.com you might name the server srv1.virtualmin.com or ns1.virtualmin.com. What name you choose is unimportant, but it must be fully qualified, it must not match a domain you’ll be managing in Virtualmin, and it must resolve, for several mail operations to work correctly.”
        • You should not name your server the same name as something you’ll be hosting in Virtualmin. It can be literally any other fully qualified domain name.
      • Primary SSL cert for main domain - #31 by MantasU - Virtualmin - Virtualmin Community
        • It would never effect Virtualmin. Virtualmin isn’t the thing that has a problem with having multiple things with the same name. The biggest issue would be Postfix, so if you try to virtual host mail on the same name as the hostname of the system, that’s a problem (because then postfix tries to map user@domain.tld to user@domain.tld which is nonsensical). There are other implications for other services. Virtualmin is not among the service that will be confused, though.
        • But, I recommend you don’t name your server something you want a website for. Just name it anything else. You never have to think about the name again or use it for anything.
        • You never have to use it for anything. You never have to worry about getting a certificate for it. You never have to worry about whether someone gets a cert warning for it, because you never have to give out the system hostname as an address that people can connect to. It’s not the main domain.
        • Just don’t name your system some name you want to use for something in Virtualmin. It’s super simple. Don’t make your system hostname important.
        • Isn’t the hostname used for email delivery? It is used when sending mail (though it doesn’t necessarily have to be, Virtualmin supports sender-dependent maps), and you don’t need a server certificate to operate as a client, which is what happens when sending mail.
        • For receiving mail, you can use any name you want. It is never the hostname of the system (it can’t be, because all mail in Virtualmin is virtually hosted…again, if you try to virtual host a domain that is the same as the hostname of the system postfix is trying to map user@domain.tld to user@domain.tld which is nonsense).
      • DKIM - Should there be 2 domains in this box - #6 by shoulders - Virtualmin - Virtualmin Community
        • You should not name your server the same as a domain name you will be hosting mail for in Virtualmin (or otherwise virtually hosting mail for). It has some of the same words, but it’s roughly the opposite direction (receive vs. send) of what you’re saying.
        • Your server hostname probably will be somewhere in the mails you send, and it’s supposed to be. It’s how the server identifies itself to other servers.
        • Edit: The key word here is virtually or virtual. Anything in the virtual map (which is what Virtualmin is managing when you create email domains) should not be the same as the name of the server.
        • Edit2: I feel like I should explain why this is, so maybe it makes more sense. The virtual map tells Postfix, “Mail for this domain can be relayed to this server”…basically mapping mail @domain.tld to a user @ the hostname of the server. But, if the name of the server is domain.tld and you have @domain.tld in virtual, you are saying, “accept mail for @domain.tld and forward it to @domain.tld”. Now, does that make sense?
      • Postfix sender_dependent_default_transport_maps per domain outgoing IP – The System Admin’s Blog
      • Use Postfix Transport Map & Relayhost Map For Flexible Email Delivery - We can configure Postfix transport_maps and sender_dependent_relayhost_maps so that some emails are delivered relay host, other emails are sent directly to recipients.
      • sender-dependent maps = can set which IP and/or route to use for a particular domain to send email.
      • There are exceptions if you are an advanced user:
        • Virtualmin --> System Settings --> Virtualmin Configuration --> Configuration category: SSL Settings --> Setup Let's Encrypt SSL certificate for hostname
        • Manually create a virtual server but disable the mail service.
        • In future version of Virtualmin the Mail services will be disabled on a virtual server using the server's hostname.
      • Catch-all email address fails when hostname - Virtualmin - Virtualmin Community
        • You should not use a bare domain (e.g. example.tld) as the hostname of your Virtualmin server, especially if you will be hosting that same domain name within Virtualmin.
      • Does virtualmin prevent you from creating a Virtual server using the hostname - Virtualmin - Virtualmin Community
        • No, but in future version of Virtualmin the mail servicce will be disabled permanently.
        • Sure, that’s fine. I just mean “you do not need to use it for anything in Virtualmin”. Not that you can’t have a sensible hostname that makes monitoring and alerting comprehensible. My point is that people keep wanting to use it for the same things that are virtually hosted in Virtualmin, which means there are two things with that name, which is a nonsensical thing to do. I think it’s just a conceptual leap that folks aren’t making; what you do in Virtualmin is virtual, it is not the physical host. Mail in Virtualmin is configured in the virtual map in Postfix. Websites configured in VirtualHost sections in Apache configuration. The system hostname is the system itself, and not anything virtually hosted on it.
      • Host default domain: SSL certificate and mail-related features - Virtualmin - Virtualmin Community - This has a discussion about this subject between the developers.
      • Suggestion to have option to set Cloudflare ports for Webmin and Usermin during automated install script - #20 by Joe - Virtualmin - Virtualmin Community - This has a discussion about this subject between the developers.
  • Choose your Primary Domain Name (eg: example.com)
    • This is domain name of the virtual server that you will setup with your hosting website, WHMCS, CRM, Client Portal, Centralised Apps or anything else related to your hosting business.
    • You can use example.com, www.example.com, anynamehere.example.com
      • These will not interfere with server.example.com as they are different domains.
      • The domain you use must be a FQDN.
    • As mentioned above, do not use your server's hostname for a virtual server as it will break the mail server.
  • Nameserver / DNS
    • Make sure your hostname and primary domain nameservers are pointing to the IP where your Virtualmin server will be.
    • You can just point A records to Virtualmin but for this tutorial it is assumed youare pointing your nameservers.
    • Don't forget that DNS changes can take up to 48 hours.
  • rDNS (reverse DNS / PTR)
    • Configure you rDNS to match what you will use for your Virtualmin's hostname eg: server.example.com
    • Not having this set correctly nowadays can lead to your email not getting delivered or at the very least sent to the SPAM folder.
    • SOLVED: Reverse DNS Does Not Match SMTP Banner in cPanel (2024) - Don't let Reverse DNS and SMTP banner mismatches in cPanel slow down your email delivery. Find the easy fix here and get back on track!

Installing

  • Downloading and Installing Virtualmin – Virtualmin - Usually, getting started with Virtualmin can be done with a few simple steps, using our automated install script. The install script will setup your package manager, usually apt-get or yum, and then download our packages as well as all of the necessary dependencies for running Virtualmin.

Post-Installation Wizard

I will give you the options I used for the wizard. The images used below show the defaults, so please read the notes for each step.

  • Introduction
  • Memory use
    • Preload Virtualmin libaries?: No
      • This alters: Virtualmin --> System Settings --> Virtualmin Configuration --> Configuration category: Server settings --> Preload Virtualmin libraries at startup:
    • Run email domain lookup server?: No
    • This step is either removed or modified to remove `Preload Virtualmin Libraries` which are now on by default
  • Virus filtering
    • Enable virus scanning with ClamAV?: Yes
      • If you do not have a lot of RAM or are not going to use email, then this is not required.
      • ClamAV currently is only used to scan emails for virus.
  • Spam filtering
    • Run SpamAssassin server filter?: No
      • This alters: Virtualmin --> Email Settings --> Spam and Virus Scanning --> SpamAssassin client program
      • Yes = spamc and no per-domain settings
        • Just your websites, small number of client hosted websites not in a strict commercially hosted enviroment and resources are limited. Global and default rules are still used.
      • No = Per-domain and Per-email address
        • You are a hosting company and each user should control their own spam.
  • Database servers

    • Run MariaDB database server?: Yes
    • Run PostgreSQL database server? No
  • MariaDB password

    • as set
  • DNS configuration

    • New Values
      • Primary nameserver: ns1.${DOM}
      • Secondary nameservers (optional): ns2.${DOM}
    • old values
      • Primary nameserver: ns1.example.com
      • Secondary nameservers (optional): ns2.example.com
    • Skip check for resolvability: ticked
  • Mandatory options all done
    • Now continue and configure the optional features.
  • Password storage
    • Password storage mode: Only store hashed password
  • MariaDB database size
    • MariaDB configuration size: use the suggested option
  • SSL key directory
    • Location for SSL certificates: Per-domain directory under /etc/ssl/virtualmin
      • Using letsencrypt by default - Webmin - Virtualmin Community
        • Q: Is it the classic way to do?
        • A: Indeed, Virtualmin defaults to storing virtual server SSL certificates in the /etc/ssl/virtualmin directory. This setup safeguards against accidental deletion of SSL certificates by users from their home directories, which could otherwise cause the webserver to fail to start.
  • Additional options all done
    • You have now completed all of the POST-Installation options.

Housekeeping

  • Delete ./root/install.sh
  • Make your root and primary user have very strong passwords.
  • Disable Webmin `root` user

Server Templates

These are used for the intial build of a Virtual Server and various POST processes such as creating a database and resetting DNS Zones. Changes are not actively reflected to accounts using the template.

Templates can be found here: Virtualmin --> System Settings --> Server Templates

This is how I have setup my templates. I will have internal websites and client websites so will need to be setup appropriately as shown below:

  • Default Settings = The default template for top-level virtual servers.
  • Settings For Sub-Servers = This is a pre-configured template for Sub-Servers which cannot be deleted.
  • Internal = This top-level template will be used for my internal websites where I want all the modern technlogies runnning.
  • Clients = This top-level template will be used for my clients who just want their websites to work and are not be bothered about advanced things such as DNSSEC and DMARC.

Configure 'Default Settings' template

Virtualmin --> System Settings --> Server Templates --> 'Default Settings' --> Edit template section:

  • Basic settings and usage
  • Administration user
    • Chroot jail new domain Unix users: No --> Yes
  • Home directory
    • Substitute variables in contents: No --> Yes
  • DNS domain
    • Primary DNS server hostname: ns1.example.com --> ns1.${DOM}
      • Default will be the primary nameserver you set during the Post-Installation Wizard.
      • If this is already set correctly then you cal leave it as is
    • Additional manually configured nameservers: .... --> ns2.${DOM}
      • Default will be the additional namerservers you set during the Post-Installation Wizard.
      • If this is already set correctly then you cal leave it as is
    • Add NS record for this system: Yes
    • Create A records for NS entries in server's domain: No --> Yes
    • Add system and virtual server's IP addresses? Ticked --> Unticked
      • This stops you internal IP getting added to your SPF records.
    • Action for other senders: Discourage (~all) --> Disallow (-all)
    • Virtualmin --> System Settings --> Server Templates --> Default Settings --> Edit template section: DNS domain --> Add sub-domain DNS records to parent domain: yes
    • TLSA: enabled when it is added
  • Mail for domain
  • Website for domain
    • Directives and settings for new websites:
      • Remove index.php4 and index.php5 if present from the DirectoryIndex statement. These have been removed from the default template in new versions of Virtualmin.
         
        DirectoryIndex index.php index.php4 index.php5 index.htm index.html

-->

DirectoryIndex index.php index.htm index.html
      • If your virtual server is aleady created you need to edit these 2 locations 
        Virtualmin --> Web configuration --> Configure SSL Website --> Edit Directives
Virtualmin --> Web configuration --> Configure Website --> Edit Directives
    • CGI script execution mode: CGI scripts disabled
      • Only enable this if you know what it is and why you want it.
      • Default is suEXEC wrapper.
      • CGI/FasCGI scripts are now a legacy technology.
    • Port number for virtual hosts: 80
    • Port number for SSL virtual hosts: 433
    • Enable HTTP2 protocol for new websites: still on default = on
    • Redirect all HTTP requests to HTTPS: unticked --> Ticked
      • let your CMS or user via the htaccess handle this.
  • SSL website for domain
  • Log file rotation
  • MariaDB database
    • Default database name: ${PREFIX} --> ${USER}_${PREFIX}
      • This setting is used if creating an intitial database when the virtual sever is created (Create database as well as login).
      • ${USER}_ Keeps naming database convention inline with cPanel and easy to understand
      • ${PREFIX} makes sure the database name is unique
      • The reason you can not just have ${USER} here is because if you have subservers this would break things or at least stop a database being created as a database of the same name might have already been created.
    • Prefix for additional databases: None --> ${USER}_
      • ${USER}_ Keeps naming database convention inline with cPanel and easy to understand
    • Create database as well as login: Yes --> No
      • This setting controls whether or not a database is created when the Virtual Server is created, and it follows the naming rule above (Default database name).
    • Default database character set: <MariaDB default> --> utf8mb4 (UTF-8 Unicode (utf8mb4))
    • Default database collation order: <MariaDB default> --> utf8mb4_unicode_ci
  • PostgreSQL (this section might not be present)
  • ProFTPD virtual FTP
  • Spam filtering
  • Webmin login
  • Virtual IP address
  • Virtual server creation
  • Plugin options
  • Default script installers
  • Mail client auto-configuration
  • PHP options
    • Default PHP version: Highest available
    • PHP configuration variables for scripts: none
      • Example:
        • PHP variable name: memory_limit
        • Comparison: At least
        • Value for variable: 32M
      • Tooltip:
        • This table can be used to enter PHP configuration settings that will be added to the FPM config or php.ini for all new virtual servers.
        • It can be useful for increasing memory limits or making other site-specific PHP config changes to satisfy application requirements.
  • Administrator's Webmin modules
    • PostgreSQL Database Server (for database): Yes --> No
    • Change Password: User password --> User and mailbox passwords
    • AWStats Reporting (for viewing reports): No --> Yes
  • New mailbox email
    • Send email to: User's mailbox --> User's mailbox + Virtual server owner
  • Updated mailbox email
    • Send email to: User's mailbox --> User's mailbox + Virtual server owner

Setup your Custom Server Templates

Virtualmin --> System Settings --> Server Templates --> Create an empty template

  • I have only included sections where you need to make changes, leave the rest on default or 'as is'.
  • E.g. Internal, Clients
  • When making your new template, select 'Default for everything' except that which you want to change. If you choose 'Create an empty template' create a blank one, this is exactly what will happen.

Internal

These are the differences from the 'Default Settings' server template.

  • Basic settings and usage
    • Template name: Internal
  • DNS domain
    • Add system and virtual server's IP addresses?: ticked --> unticked
    • Add DMARC DNS record: Yes, with policy below
    • DMARC policy for emails that fail SPF or DKIM: Reject Email
    • Create DNSSEC key and sign new domains: Yes
    • DNSSEC cryptographic algorithm: ECDSAP256SHA256
  • PHP options
    • Default PHP execution mode: FPM

Clients

These are the differences from the 'Default Settings' server template.

  • Basic settings and usage
    • Template name: Clients
  • PHP options
    • Default PHP execution mode: FPM
      • it is set to mod_php because of a bug.

Setup your Sub-Server Template

These are used for setting up sub-servers and their options are inherited from 'Default Settings' template, not the parent's template.

This system is not ideal and might get some inprovement, however the 'Settings For Sub-Servers' template does not need much altering for most people at this time. If you did need to make any changes I would recommend copying this template and name it to match the top-level server templates they will be used in conjuction with (ie. Internal, Client).

Sub-Server templates only really work if they do not have mail, and the DNS is managed by the parent so the inheritance from the 'Default Settings' rather that the parent's template does not become an issue. When it does matter you must make copies of the 'Settings For Sub-Servers' template and work on that instead of a single template for Sub-Servers.

Settings For Sub-Servers

These are the differences from the 'Default Settings' server template.

  • DNS domain
    • Add system and virtual server's IP addresses?: ticked --> unticked
    • Add DMARC DNS record: Yes, with policy below
    • DMARC policy for emails that fail SPF or DKIM: Reject Email
    • Create DNSSEC key and sign new domains: Yes
    • DNSSEC cryptographic algorithm: ECDSAP256SHA256
  • PHP options
    • Default PHP execution mode: FPM
  • MariaDB database
    • Prefix for additional databases: From default settings
      • When the template is related to a sub-server, variables for the parent server are also available with PARENT_DOMAIN_ prefix, like ${PARENT_DOMAIN_HOME} and ${PARENT_DOMAIN_DOM}

Server Template House Keeping

Theese options now need to be set and are common to all templates.

  • Set your default Server and Sub-Server templates

Account Plans

These control things like: Permissions, Features, Bandwidth and Disk Quotas.

  • Setup your Account Plans
    • Virtualmin --> System Settings --> Account Plans --> Add a new account plan
      • Examples
        • Primary(5000Mib)
        • Internal(Unlimited)
        • Bronze(1000MiB)
        • Silver(1500MiB)
        • Gold (2000MiB)
      • Allowed virtual server features
        • Automatic, based on initial features
          • Do not change the services that were created at setup (enabled/disabled status), leave them as they are.
          • Not really automatic.
      • Allowed capabilites
        • Automatic, based on other limits
          • This is ok for your clients but can be restrictive. Always check with a dummy account the options are suitable.
        • Selected below
          • Internal Account Plan manual settings - These might be alright for your clients if they have some IT experience.
  • Set Default Account Plan
    • Virtualmin --> System Settings --> Account Plans --> Set default plan to: Bronze

Services (Daemons)

Apache

  • Enable the following Apache modules
    • Webmin --> Servers --> Apache Webserver --> Global configuration --> Configure Apache Modules
    • brotli
      • This requires more than enabling the apache module.
      • Gains over gzip/deflate are not massive.
    • expires
    • headers
    • You must restart the Apache server for the changes to be reflected.
      • Webmin --> System --> Bootup and Shutdown --> apache2.service
  • Add the following recommended security headers (from wordpress, you do these from the CMS not Apache)

ProFTPd (FTP)

  • Set the default transfer mode to Binary
    • Webmin --> Servers --> ProFTPD Server --> Networking Options --> Default transfer mode: Binary
    • DefaultTransferMode - ProFTPD module mod_xfer | proftpd.org
      • Default: ascii
      • The DefaultTransferMode directive sets the default transfer mode used for data transfers.
      • Per RFC 959 requirements, the default transfer mode is "ascii", which means that carriage return (CR) and line feed (LF) translation will be performed: CRLF sequences in uploaded data will be translated to LF, and LF translated to CRLF in downloaded data.
  • Disable Symbolic Links (optional)
    • Webmin --> Servers --> ProFTPD --> Files and Directories --> Show symbolic links: No
    • This will prevent accessing AWStats via FTP.
  • Force TLS on FTP
    • This currently does not have any options in the GUI to enable this, but can be done by modifying the config files.
    • Edit the config file - Webmin -->Servers --> ProFTPD Server --> Edit Config Files --> Editing Directives in File: /etc/proftpd/conf.d/virtualmin.conf
      • Enforce TLS by adding: 
        TLSRequired                   off

-->

TLSRequired                   on
      • Add the following to declare what TLS protocols are allowed just below 'TLSRequired'. 
        TLSProtocol                   TLSv1.2 TLSv1.3
        • The protocols have to be installed on the system to work.
      • Save the config.
    • Apply the changes (this will restart the ProFTPD service).
    • ProFTPD: FTP and SSL/TLS | proftp.org - The mod_tls module for proftpd is an implementation of RFC 4217.

PHP

  • Install additional PHP versions as required
  • Configure the values in the global php.ini files for each version of PHP version as required
    • Webmin --> Tools --> PHP Configuration
    • disable_functions: 
      # Short Version
disable_functions = system,passthru,popen,exec,proc_close,proc_get_status,proc_nice,proc_open,proc_terminate,highlight_file,escapeshellcmd,define_syslog_variables,posix_uname,posix_getpwuid,apache_child_terminate,posix_kill,posix_mkfifo,posix_setpgid,posix_setsid,posix_setuid,escapeshellarg,posix_uname,ftp_exec,ftp_connect,ftp_login,ftp_get,ftp_put,ftp_nb_fput,ftp_raw,ftp_rawlist,ini_alter,ini_restore,inject_code,syslog,openlog,define_syslog_variables,apache_setenv,mysql_pconnect,eval,phpAds_XmlRpc,phpAds_remoteInfo,phpAds_xmlrpcEncode,phpAds_xmlrpcDecode,xmlrpc_entity_decode,fp,fput,shell_exec,apache_get_modulesi,

# Default Virtualmin 7.4 FPM
disable_functions = pcntl_alarm,pcntl_fork,pcntl_waitpid,pcntl_wait,pcntl_wifexited,pcntl_wifstopped,pcntl_wifsignaled,pcntl_wifcontinued,pcntl_wexitstatus,pcntl_wtermsig,pcntl_wstopsig,pcntl_signal,pcntl_signal_get_handler,pcntl_signal_dispatch,pcntl_get_last_error,pcntl_strerror,pcntl_sigprocmask,pcntl_sigwaitinfo,pcntl_sigtimedwait,pcntl_exec,pcntl_getpriority,pcntl_setpriority,pcntl_async_signals,pcntl_unshare,

# Combined Version
disable_functions = pcntl_alarm,pcntl_fork,pcntl_waitpid,pcntl_wait,pcntl_wifexited,pcntl_wifstopped,pcntl_wifsignaled,pcntl_wifcontinued,pcntl_wexitstatus,pcntl_wtermsig,pcntl_wstopsig,pcntl_signal,pcntl_signal_get_handler,pcntl_signal_dispatch,pcntl_get_last_error,pcntl_strerror,pcntl_sigprocmask,pcntl_sigwaitinfo,pcntl_sigtimedwait,pcntl_exec,pcntl_getpriority,pcntl_setpriority,pcntl_async_signals,pcntl_unshare,system,passthru,popen,exec,proc_close,proc_get_status,proc_nice,proc_open,proc_terminate,highlight_file,escapeshellcmd,define_syslog_variables,posix_uname,posix_getpwuid,apache_child_terminate,posix_kill,posix_mkfifo,posix_setpgid,posix_setsid,posix_setuid,escapeshellarg,posix_uname,ftp_exec,ftp_connect,ftp_login,ftp_get,ftp_put,ftp_nb_fput,ftp_raw,ftp_rawlist,ini_alter,ini_restore,inject_code,syslog,openlog,define_syslog_variables,apache_setenv,mysql_pconnect,eval,phpAds_XmlRpc,phpAds_remoteInfo,phpAds_xmlrpcEncode,phpAds_xmlrpcDecode,xmlrpc_entity_decode,fp,fput,shell_exec,apache_get_modulesi,

NB: mail can be added to prevent the use of mail()
    • You must restart the PHP services for the changes to be reflected.
      • Webmin --> System --> Bootup and Shutdown --> php*
  • Install missing PHP modules
    • Via either of these methods
      1. Command Line 
        apt-get install php-{curl,gd,imagick,intl,zip}
        • Notice the php version number has been removed
      2. Webmin --> System --> Software Package Update --> Only new
        • php-curl
        • php-gd
        • php-imagick (this might need ImageMagic installing)
        • php-intl
        • php-zip
    • NB: The MultiPHP install commandline will have a more complete list of extensions to install.

MariaDB (Database)

nothing to change

BIND (DNS)

systemd-resolved (DNS Stub Resolver)

This is the server working as a client, not a server, see BIND section above for those settings.

This section is based heavily on my Ubuntu server setup but the issues and solutions might apply to other flavours of Linux.

Enable DNS over TLS (DoT)

systemd-resolved can be configured to handle it's requests over an encrypted connection which is better for security as you cannot perform a MITM attack. This should be enabled even if you are just using a local network becasue once an attacker is in your network he could otherwise harvest all of this data.

  • Instructions 
    ## Configure the connection mode
edit /etc/systemd/resolved.conf
#DNSOverTLS=no --> DNSOverTLS=opportunistic

## Restart service
systemctl restart systemd-resolved

## Check `systemd-resolved` for DoT support
resolvectl status

## Ping an external site to verify DNS is working
ping virtualmin.com
  • DNSOverTLS modes
  • Misc
  • Man Pages
    • systemd(1) — Arch manual pages - systemd is a system and service manager for Linux operating systems.
    • resolvconf.conf(5) — Arch manual pages - resolvconf configuration file
    • resolvectl(1) — Arch manual pages - resolvectl may be used to resolve domain names.
    • DNS over TLS - systemd-resolved - ArchWiki
      • systemd-resolved is a systemd service that provides network name resolution to local applications via a D-Bus interface, the resolve NSS service (nss-resolve), and a local DNS stub listener on 127.0.0.53.
      • DNS over TLS is disabled by default. To enable it change the DNSOverTLS setting in the [Resolve] section in resolved.conf(5).
      • To enable validation of your DNS provider's server certificate, include their hostname in the DNS setting in the format ip_address#hostname. For example......
    • resolved.conf(5) - Linux manual page - These configuration files control local DNS and LLMNR name resolution.
    • ENVIRONMENT - systemd(1) — Arch manual pages
      • Denotes system log levels thart can e usied in various related utilities.
      • $SYSTEMD_LOG_LEVEL
        • The maximum log level of emitted messages (messages with a higher log level, i.e. less important ones, will be suppressed). Takes a comma-separated list of values. A value may be either one of (in order of decreasing importance) emerg, alert, crit, err, warning, notice, info, debug, or an integer in the range 0...7. See syslog(3) for more information.
  • Troubleshooting (DNS over TLS / DoT)
    • Port Checker - Check Open Ports Online - Port Checker is a simple tool to check for open ports and test port forwarding setup on your router. Verify and diagnose connection errors on your computer.
    • Monitor the log for DNS queries
      ## Check logging level
resolvectl log-level

## Set loggin level to debug
resolvectl log-level debug

## Live monitor the log
journalctl -u systemd-resolved -f

## Run a DNS query
resolvectl query bbc.co.uk

# Revert the log level (info is default level and reverts after a restrt of the service)
resolvectl log-level info
    • Logging (General)
    • Failed to invoke gnutls_handshake: Error in the certificate verification. 
      ## Error on the command line
root@server:~# resolvectl query bbc.co.uk
bbc.co.uk: resolve call failed: All attempts to contact name servers or networks failed
root@server:~#         

-->

## Errors in the logLoggin errors
Jul 22 14:36:37 devweb.svchost.uk systemd-resolved[84868]: Failed to invoke gnutls_handshake: Error in the certificate verification.
Jul 22 14:36:37 devweb.svchost.uk systemd-resolved[84868]: Connection failure for DNS TCP stream: Connection refused
      • This error is caused by the certificate of the remote DNS server not being validated, either because it has expired, broken or is a self-signed certificate etc... and you are on DNSOverTLS=yes which does not allow anything but a strict and valid TLS certificate and chain with no downgrade capability.
    • DNS_over_TLS - systemd-resolved - ArchWiki
      • ngrep can be used to test if DNS over TLS is working since DNS over TLS always uses port 853 and never port 53.
      • The command ngrep port 53 should produce no output when a hostname is resolved with DNS over TLS and ngrep port 853 should produce encrypted output.
      • If you have no traffic
        • in one terminal run the ngrep command,
        • in another run a DNS query command of your choosing, you can even run ping.
      • If you have configure DoT correctly you should see no traffic when you ngrep port 53.
    • How to troubleshoot DNS with systemd-resolved? - Unix & Linux Stack Exchange - How would you go about finding the DNS servers used by systemd-resolved, for troubleshooting purposes?
Enable DNSSEC Support (as client) / Preparing for DANE

All local DNS requests are resolved by the systemd-resolved local stub resolver on 127.0.0.53 which by default does not handle DNSSEC (bits) on the DNS requests and therefore cannot validate domains via DNSSEC for the various apps and CLI that call it.

DNSSEC technology should be used if it is easy to enable, which it is.

Install DIG and Delv

It is perfectly safe to install these dnsutils on your live server.

Some flavours of Linux will have these utilties already installed but when your base OS is Ubuntu Server minimised you will find a lot of utilities are not installed which is normal, however this means we will need to install the Bind9 Utilities package dnsutils as follows:

sudo apt install dnsutils

or you can use : Webmin --> System --> Software Packages

Test if DNSSEC is working

Below are variety of different test you can use to verify DNSSEC capabilities

### Check DNSSEC Support

## Check `systemd-resolved` for DNSSEC support
resolvectl status

## Flush Cache
resolvectl flush-caches

### DNSSEC Validation Tests

## Dig
dig sigok.verteiltesysteme.net
dig sigok.verteiltesysteme.net @10.0.0.1 +dnssec 
dig sigok.verteiltesysteme.net @10.0.0.1 +dnssec | grep status
dig sigfail.verteiltesysteme.net
dig sigfail.verteiltesysteme.net @10.0.0.1 +dnssec
dig sigfail.verteiltesysteme.net @10.0.0.1 +dnssec | grep status
dig example.com DS
dig example.com DNSKEY +dnssec +cd +multiline 

## Delv
delv example.com
delv example.com @10.0.0.1 +dnssec
delv example.com soa +multi     (option that formats large records into multiline reports that are readable in a standard 80-column text window. )
delv example.com soa +multi -i  (-i = disables DNSSEC validatin)
delv example.com +multi +vtrace (+vtrace option shows the entire DNSSEC chain of validation.)
delv example.com +multi +rtrace (+rtrace prints the extra DNS lookups that delv needs to make while validating the reply to a query. )

## Resolvectl
resolvectl query sigok.verteiltesysteme.net
resolvectl query sigfail.verteiltesysteme.net

### Domains

## Good Domains
sigok.verteiltesysteme.net
go.dnscheck.tools
internetsociety.org
dnssec-tools.org
dnssec-deployment.org

## Bad Domains
sigfail.verteiltesysteme.net
badsig.go.dnscheck.tools
brokendnssec.net
dnssec-failed.org   (operated by Comcast)
rhybar.cz           (operated by CZ.NIC)
  • dig
  • delv
  • Online Testers
    • DNSViz | A DNS visualization tool - DNSViz is a tool for visualizing the status of a DNS zone. It was designed as a resource for understanding and troubleshooting deployment of the DNS Security Extensions (DNSSEC). It provides a visual analysis of the DNSSEC authentication chain for a domain name and its resolution path in the DNS namespace, and it lists configuration errors detected by the tool.
    • DNSSEC Debugger (Verisign) - The DNSSEC Debugger from VeriSign Labs is an on-line tool to assist with diagnosing problems with DNSSEC-signed names and zones.
    • DNSSEC Resolver Test - This web-based test checks whether your domain name lookups are protected by DNSSEC.
  • Misc
  • Troubleshooting DNSSEC
  • Errors
    • Some domains with DNSSEC enabled cannot be resolved by my server
      • Related Errors
        • DNS Commands
           
          ## resolvectl DNS lookup (from my webserver)

root@server:~# resolvectl query ns1.example.com
ns1.example.com: resolve call failed: DNSSEC validation failed: no-signature
root@server:~# 


## delv DNS lookup (from my pfsense router)

[2.7.2-RELEASE][admin@pfs.example.com]/root: delv ns1.example.com soa +multi
;; insecurity proof failed resolving 'example.com/DNSKEY/IN': 127.0.0.1#53
;;   validating XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.example.com/NSEC3: bad cache hit (example.com/DNSKEY)
;; broken trust chain resolving 'ns1.example.com/SOA/IN': 127.0.0.1#53
;; resolution failed: broken trust chain

[2.7.2-RELEASE][admin@pfs.example.com]/root: delv example.com soa +multi
;; insecurity proof failed resolving 'example.com/SOA/IN': 127.0.0.1#53
;; resolution failed: insecurity proof failed
[2.7.2-RELEASE][admin@pfs.example.com]/root:
        • resolve call failed: DNSSEC validation failed: no-signature
        • resolution failed: broken trust chain
        • resolution failed: insecurity proof failed
        • Virtualmin
          • Warning! Errors were found in this domain's DNS records : NS record ns1.example.com. cannot be resolved to an IP address
      • Background:
        • I enabled DNSSEC validation on my systemd-resolved stub resolver which means if a domain has DNSSEC then it should be correct or the lookup will fail.
        • Clients on external networks have no issue.
        • Local DNS lookups on without DNSSEC validation work normally.
        • External DNSSEC request are working fine
        • DNSSEC chain of trust is all valid
        • DNSSEC validation is successful on DNSViz and DNSSEC Debugger | Verisign
      • Issue:
        • For one or more domain(s) on your webserver with DNSSEC enabled:
          • Local DNS lookups are failing
          • Ping will not resolve these domain names
          • DNSSEC validation fails
      • Cause:
        • A combination of:
          • Enforcing DNSSEC validated DNS lookups (This is webservers DNS lookup mechanism, not BIND)
            • on systemd-resolved DNS Stub Resolver
          • Split DNS
      • Explanation:
        • When using split DNS for your webserver, DNSSEC validation fails because your local IP is used for your webserver rather than your public IP causing the trust chain to be broken.
          • i.e. Your webserver's local IP is defined in your routers host list against certain domains, causing DNS lookups for these domains to return this local IP in DNS lookups for said daomins, which in turn causes the client to use this IP for accessing the websites and other domain related taks such as ping.
        • DNS lookups on domain's that have a local IP will always fail DNSSEC valiation, so these lookups will always fail when DNSSEC validation is enabled.
        • The whole purpose of DNSSEC is to make sure the DNS lookup returns the correct IPs for the domain requested in a cryptographically secure manner which makes this behaviour completely normal as the local IP is not a valid IP for the chain of trust, only Your public IP is the correct IP for the domain.
      • Solution:
        • Disable Split DNS for the offending domain.
          • This will be handled in your router, in my case pfSense.
        • Make sure you have NAT reflection enabled so you can access your webserver from the internal network via your public IP.
          • This is not required to fix the validation error but is a remedy to allow access to your websites/webserver as normal from your locall network.
          • This will be handled in your router, in my case pfSense.
    • DNSSEC Validation - Cookie Warnings on DNSviz
      • These are warnings and do not affect validation
      • You will see them easily here: DNSViz | A DNS visualization tool
      • The errors

        ./DNSKEY: The server appears to support DNS cookies but did not return a COOKIE option. See RFC 7873, Sec. 5.2.3. (2001:500:2f::f, UDP_-_EDNS0_512_D_KN)

uk/DS (alg 8, id 43876): The server appears to support DNS cookies but did not return a COOKIE option. See RFC 7873, Sec. 5.2.3. (2001:500:2f::f, UDP_-_EDNS0_4096_D_KN)

uk/DS (alg 8, id 43876): The server appears to support DNS cookies but did not return a COOKIE option. See RFC 7873, Sec. 5.2.3. (2001:500:2f::f, UDP_-_EDNS0_4096_D_KN)
      • RFC 7873: Domain Name System (DNS) Cookies
Enable DNSSEC - Different Options

So if DNSSEC is not enabled, you now need to set it up following the instructions below.

Consider your network infrastructure such as you have a pfSense router as a fully configures DNS resolver you would not need a local DNS cache on your Ubuntu server but if you were just using publically availabe DNS servers such as 1.1.1.1, 8.8.8.8 and 9.9.9.9 then you would need a local cache.

  • In pfSense:
    • Services --> DNS Resolver --> General Settings --> General DNS Resolver Options --> DNSSEC:
      • This option does not need to be on for systemd-resolved to handle DNSSEC requests and responses.
      • pfSense forwards all DNS traffic upstream and caches as required.
      • This assumes you have setup pfSense according to my tutorial. eg:
        • Services --> DNS Resolver --> General Settings --> General DNS Resolver Options --> DNS Query Forwarding: ticked

Option 1 - Enable DNSSEC support in `systemd-resolved` (Preferred)

  • Why choose this option
    • Simple to implement, no big system changes, has parity with how a Window slocal DNS cache works
  • Pro
    • This is the same setup as you have now but with DNSSEC support
    • easy to implement
    • you don't need to change any system files
    • systemd-resolved is still running
    • you can make use of the local DNS caching
  • Con
    • DNS lookup statistics on your router (DNS Server) will not match up becasue of the local caching.
  •  Instructions 
    ## Enable DNSSEC in systemd-resolved
edit /etc/systemd/resolved.conf
#DNSSEC=no --> DNSSEC=yes

## Restart service
systemctl restart systemd-resolved

## Check `systemd-resolved` for DNSSEC support
resolvectl status

Option 2 - Disable `systemd-resolved`stub DNS resolver service and use only an external DNS server that supports DNSSEC

  • Why choose this option
    • This is ideal if you have a local DNS server (same network, different machine) with caching and all those other policy things (i.e. pfSense).
  •  Pro
    • You don't need to change any system files
    • DNS lookup statistics on your router (DNS Server) will be correct and will allow for deeper analysis
    • The rest of the `systemd-resolved` system stays in place
  • Con
    • Local caching is disabled
    • Compatibility with traditional Linux programs might be affected if the /etc/resolv.conf is no longer symlinked to a valid configuration.
      • This might not be true becauevI think the symlink is still present but just the listener service bound on port 53 is removed and corresponding refernces in the config and NIC.
  • Instructions 
    ## Disable DNS Stub Listener
edit /etc/systemd/resolved.conf
#DNSStubListener=yes --> DNSStubListener=no

## Restart service
systemctl restart systemd-resolved

## Configure NIC
Remove 127.0.0.53 if present
Add new DNS server (i.e. 10.0.0.1) unless it is aquired by DHCP

Option 3 - remove resolv.conf symlink and use your own

  • Why choose this option
    • Developers might use this
    • Might have some legacy application uses
  • Pro
    • You gain manual control of the resolution system for legacy apps and CLI.
    • DNS lookup statistics on your router (DNS Server) will be correct and will allow for deeper analysis
    • `systemd-resolved` is completely bypassed
  • Con
    • You have to change the system files by deleting a symlink and making your own resolve.conf file
    • This should only be done if you understand all of the consequences.
    • Local caching is disabled
  • Instructions 
    ## Make a backup of this file's contents
/etc/resolv.conf 

## Remove the symbolic link for the config file
unlink /etc/resolv.conf

## Create a static config file
touch /etc/resolv.conf

## Edit the file as required (this is just an example)
  nameserver 10.0.0.1
  options edns0 trust-ad
  search .

## Restart service
systemctl restart systemd-resolved

## Configure NIC
Remove 127.0.0.53 if present
Add new DNS server (i.e. 10.0.0.1) unless it is aquired by DHCP
  • Reverse the above instructions
    ## Remove the static file resolv.conf
rm -f /etc/resolv.conf

## Create a Symbolic Link between resolv.conf and the dynamically created config
ln -sv /run/systemd/resolve/resolv.conf /etc/resolv.conf

## Configure NIC
Add 127.0.0.53 to the NIC DNS as the first entry

## Restart service
systemctl restart systemd-resolved

PostFix (Email)

  • Let SpamAssassin check blocklists, not Postfix, leave this to be an MTA only.
  • All information for the commands below can be accesss by using https://www.postfix.org/postconf.5.html#YourCommandHere
  • Restrictions (values) are separated by commas and/or whitespace. Continue long lines by starting the next line with whitespace. Restrictions are applied in the order as specified; the first restriction that matches wins.
    • Separating restrictions by using a comma and space ", " is preferred as shown if you look at the live configuration using postconf -d
SMTP access restriction Policy

Remove `SMTP Client Restrictions` and create `Access restriction lists` · Issue #2171 · webmin/webmin · GitHub - This shows how I would like the Access restriction lists should be.

These settings will control how Postfix handles sending and receiving emails, i.e. allow authenticated users only, only allow emails to be sent from defiend IP addresses and checking for malformed headers etc...

I have separated out these settings because it is more logical to enter them in this order and makes it easier for you to alter them as required.

  • Postfix SMTP relay and access control | postfix.org
    • Postfix allows you to specify lists of access restrictions for each stage of the SMTP conversation. Individual restrictions are described in the postconf(5) manual page.
    • Stages are processed in order as outlined in the table.
    • Each restriction list is evaluated from left to right until some restriction produces a result of PERMIT, REJECT or DEFER (try again later). The end of each list is equivalent to a PERMIT result. By placing a PERMIT restriction before a REJECT restriction you can make exceptions for specific clients or users. This is called allowlisting; the smtpd_relay_restrictions example above allows mail from local networks, and from SASL authenticated clients, but otherwise rejects mail to arbitrary destinations.
    • If an email passes a stage it is permitted to go onto the next stage. If it fails a test (e.g rejected) the email will be failed and will not be delivered.
    • You can put `reject`at the end of a list so if non of the test are positive, the email will fail.

 

  • permit_mynetworks
    • Local clients will pass the stage, all other tests will be ignored and the process will then move onto the next stage.
    • Local clients are those user on networks as defined by the settings mynetworks and/or mynetworks_style.
    • If you just want to only allow authenticated users, just remove "permit_mynetworks" from all of the stages (or as required).
  • permit_sasl_authenticated
    • Authenticated clients will pass the stage, all other tests will be ignored and the process will then move onto the next stage.

 

  1. smtpd_client_restrictions (Client Connection Policy - Is a client allowed to connect)
    • Webmin --> Servers --> Postfix Mail Server --> SMTP Client Restrictions --> Client restrictions: 
      # These are correct but the Virtualmin GUI options do not currently match this stage, so these will need to be entered directly into the config file

permit_mynetworks, permit_sasl_authenticated, reject_unknown_client_hostname
    • Via GUI:
      • Allow connections from same network: ticked
        • (permit_mynetworks)
        • Permit the request when the client IP address matches any network or network address listed in $mynetworks.
        • You can specify the list of "trusted" network addresses by hand or you can let Postfix do it for you (which is the default). See the description of the mynetworks_style parameter for more information.
      • Allow connections from this system: unticked
        • (permit_inet_interfaces)
        • Allow connections from this system
        • Always allow email from the server to the server.
        • Permit the request when the client IP address matches $inet_interfaces.
        • I think this is useful if your server is connected to more that one network, i.e. multiple NICs.
      • Reject clients with no reverse hostname: ticked
        • (reject_unknown_reverse_client_hostname)
        • Reject clients with no reverse hostname
        • Reject the request when the client IP address has no address->name mapping.
        • This is weaker than reject_unknown_client_hostname 
      • Allow TLS clients with any certificate: unticked
        • (permit_tls_all_clientcerts)
      • Allow authenticated clients: ticked
        • (permit_sasl_authenticated)
        • Allow authenticated clients
      • Check client access map: empty
        • (check_client_access example.txt)
      • Reject if client IP address is in RBL: empty
        • (reject_rbl_client example-rbl.com)
      • Reject if client hostname is in RBL: empty
        • (reject_rhsbl_client example-rbl.com)
    • VM Default: Postfix default (allow all clients) + all empty
    • No tooltip
    • Optional restrictions that the Postfix SMTP server applies in the context of a client connection request.
    • reject_unknown_client_hostname
      • You would replace reject_unknown_reverse_client_hostname with this for stonger protection.
      • Reject the request when
        • the client IP address->name mapping fails, or
        • the name->address mapping fails, or
        • the name->address mapping does not match the client IP address.
      • This is stronger than reject_unknown_reverse_client_hostname.
      • This can only be swapped by editing the config file because the GUI does not have the ability.
    • Reject policy 
      , reject
      • If you add `, reject` at the end of the policy you will restrict connections to your email server to local clients and authenticated clients only.
      • Don't add this unless you really want to lock down you server to your company's clients and servers.
      • This will stop external email servers connecting to your server to deliver emails unless you also define exceptions.
      • This is an example error you will get when an external email server sends an email while the `reject` is on:
         
        May 25 11:54:25 server.example.com postfix/anvil[836633]: statistics: max cache size 1 at May 25 11:51:05
May 25 11:54:25 server.example.com postfix/anvil[836633]: statistics: max connection count 1 for (smtp:44.44.44.44) at May 25 11:51:05
May 25 11:54:25 server.example.com postfix/anvil[836633]: statistics: max connection rate 1/60s for (smtp:44.44.44.44) at May 25 11:51:05
May 25 11:51:05 server.example.com postfix/smtpd[836630]: disconnect from server.remotehost.com[44.44.44.44] ehlo=2 starttls=1 mail=1 rcpt=0/1 data=0/1 quit=1 commands=5/7
May 25 11:51:05 server.example.com postfix/smtpd[836630]: NOQUEUE: reject: RCPT from server.remotehost.com[44.44.44.44]: 554 5.7.1 <server.remotehost.com[44.44.44.44]>: Client host rejected: Access denied; from=<bob@remotewebsite.com> to=<test@example.com> proto=ESMTP helo=<server.remotehost.com>
May 25 11:51:05 server.example.com milter-greylist[632]: GeoIP is not available
May 25 11:51:05 server.example.com postfix/smtpd[836630]: connect from server.remotehost.com[44.44.44.44]
    • the virtualmin implementation is wrong, the variable options do not match the stage
    • The code in the <pre> box is correct
  2. smtpd_helo_restrictions (HELO Handshake Policy - Don't talk to mail systems that don't know their own hostname or have an invalid HELO / ELO)
    • Webmin --> Servers --> Postfix Mail Server --> SMTP Server Options --> Restrictions on sends in HELO commands: 
      permit_mynetworks, permit_sasl_authenticated, reject_invalid_helo_hostname, reject_non_fqdn_helo_hostname, reject_unknown_helo_hostname
    • VM Default: Default
    • No tooltip
    • Optional restrictions that the Postfix SMTP server applies in the context of a client HELO command.
    • he HELO handshake is handled by the server (hostname) and not the particular website or domain.
  3. smtpd_sender_restrictions (Sender Policy - Don't accept mail from dodgy domains / domains that don't exist)
    • Webmin --> Servers --> Postfix Mail Server --> SMTP Server Options --> Restrictions on sender addresses: 
      reject_non_fqdn_sender, reject_unknown_sender_domain
    • Default: Default
    • Optional restrictions that the Postfix SMTP server applies in the context of a client MAIL FROM command.
    • These restrictions are specific to the sender address received with the MAIL FROM command.
    • It's generally polite to say who the mail is from. Again, very few real mail do not have a return address, most who don't are spam.
    • reject_non_fqdn_sender
      • Reject mail which doesn't have a valid to and from domain:
    • reject_unknown_sender_domain
      • Reject mail where there is no known sender domain:
  4. smtpd_recipient_restrictions (Spam Blocking Policy - Check the email for spam indicators, you would put RBL lookups here)
    • Webmin --> Servers --> Postfix Mail Server --> SMTP Server Options --> Restrictions on recipient addresses: 
      reject_non_fqdn_recipient, reject_unknown_recipient_domain
    • Default: permit_mynetworks permit_sasl_authenticated reject_unauth_destination
    • Optional restrictions that the Postfix SMTP server applies in the context of a client RCPT TO command, after smtpd_relay_restrictions.
    • I have not added any RBLs because my firewall block IPs and SpamAssassin should handle spam.
    • These restrictions are specific to the recipient address that is received with the RCPT TO command.
    • reject_non_fqdn_recipient
      • Reject mail which doesn't have a valid to and from domain:
    • reject_unknown_recipient_domain
      • Reject mail where there is no known reciever domain:
    • RBL Examples
      • You would just add these lines in your `smtpd_recipient_restrictions` setting. Other services are available.
        reject_rbl_client zen.spamhaus.org, reject_rhsbl_reverse_client dbl.spamhaus.org, reject_rhsbl_helo dbl.spamhaus.org, reject_rhsbl_sender dbl.spamhaus.org
    • Postgrey
      • This gets added when you enable Greylisting 
        check_policy_service inet:127.0.0.1:10023
    • ---------Other location---------
      • Webmin --> Servers --> Postfix Mail Server --> SMTP Authentication And Encryption --> SMTP recipient restrictions
        • Dont use or change values here - reported here Postfix - `permit_mx_backup ` and `check_relay_domains` should be removed · Issue #2150 · webmin/webmin · GitHub
        • Default ticked:
          • Allow connections from same network
          • Allow authenticated clients
          • Reject email to other domains
        • No tooltip
        • Alters same config as: Webmin --> Servers --> Postfix Mail Server --> SMTP Server Options --> Restrictions on recipient addresses:
        • These are the option mappings from the GUI
          • Allow connections from same network = permit_mynetworks (smtpd_client_restrictions)
          • Allow connections from this system = permit_inet_interfaces (smtpd_client_restrictions)
          • Reject clients with no reverse hostname = reject_unknown_reverse_client_hostname (smtpd_client_restrictions)
          • Allow authenticated clients = permit_sasl_authenticated (smtpd_client_restrictions)
          • Reject email to other domains = reject_unauth_destination (smtpd_recipient_restrictions)
          • Allow only relay domains = check_relay_domains = removed (remove)
          • Allow domains this system is a backup MX for= permit_mx_backup = is going to be removed (remove)
          • not for single server setup
  5. smtpd_relay_restrictions (Relay Policy - Control who can send emails where, covering internal/external network/server/domain endpoints - who can send emails in and out of our server's control)
    • Webmin --> Servers --> Postfix Mail Server --> SMTP Authentication And Encryption --> SMTP relay restrictions: 
      # These are not the same options as below because some options are not currently available in the GUI and need to be added directly into the config file

permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination
    • Via the GUI, select as follows (read above first):
      • Allow connections from same network: ticked
        • (permit_mynetworks)
        • wrong contect - should be in sm
      • Allow connections from this system: unticked
        • (permit_inet_interfaces)
      • Reject clients with no reverse hostname:ticked
        • (reject_unknown_reverse_client_hostname)
      • Allow authenticated clients: ticked
        • (permit_sasl_authenticated)
      • Reject email to other domains: ticked
        • (reject_unauth_destination)
      • Allow only relay domains: unticked
        • (check_relay_domains)
      • Allow domains this system is a backup MX for: unticked
        • (permit_mx_backup)
      • Default ticked:
        • Allow connections from same network
        • Allow authenticated clients
      • No tooltip
      • Access restrictions for mail relay control that the Postfix SMTP server applies in the context of the RCPT TO command, before smtpd_recipient_restrictions.
      • Because of the rigid text boxes not all options can configured through the GUI.
      • The same restrictions are available as documented under smtpd_recipient_restrictions.
    • Local clients and authenticated clients may still specify any destination domain.........
    • reject_unauth_destination
      • Checks the email destination resolves correctly with no unexpected routing.
      • It tells postfix not to accept messages with recipients at domains that are not hosted locally or that we serve as a backup server for. Without this line, our server would be an open relay.
    • Reject policy 
      , reject
      • If you add `, reject` at the end of the policy you will restrict relaying email server to local clients and authenticated clients only.
      • Don't add this unless you really want to lock down you server to your company's clients and servers.
      • This will stop external email servers relaying emails to your server unless you also define exceptions.
  6. smtpd_data_restrictions (Data Policy - I think: this acts during the DATA transfer) (not in virtualmin GUI)
    • The Code 
      reject_unauth_pipelining
    • `reject_unauth_pipelining` blocks clients that speak too early.
    • Optional access restrictions that the Postfix SMTP server applies in the context of the SMTP DATA command. See SMTPD_ACCESS_README, section "Delayed evaluation of SMTP access restriction lists" for a discussion of evaluation context and time.
    • reject_unauth_pipelining
      • Reject the request when the client sends SMTP commands ahead of time where it is not allowed, or when the client sends SMTP commands ahead of time without knowing that Postfix actually supports ESMTP command pipelining. This stops mail from bulk mail software that improperly uses ESMTP command pipelining in order to speed up deliveries.
      • With Postfix 2.6 and later, the SMTP server sets a per-session flag whenever it detects illegal pipelining, including pipelined HELO or EHLO commands. The reject_unauth_pipelining feature simply tests whether the flag was set at any point in time during the session.
      • Postfix supports a technique known as pipelining that speeds up bulk deliveries of email by sending multiple smtp commands at once. The protocol requires that clients first check that the server supports pipelining. Many spammers send a series of commands without waiting for authorization, in order to deliver their messages as quickly as possible. reject_unauth_pipelining stops mail from bulk mail software that improperly uses pipelining in order to speed up deliveries.
    • Postfix - missing restrictions - `smtpd_data_restrictions` and `smtpd_end_of_data_restrictions` · Issue #2167 · webmin/webmin · GitHub
  7. smtpd_end_of_data_restrictions (After Data Policy - I think: this is triggered just after DATA payload is received) (not in virtualmin GUI)
  8. smtpd_etrn_restrictions (After Email Policy - I think: After the email is fully received)
    • Webmin --> Servers --> Postfix Mail Server --> SMTP Server Options --> Restrict ETRN command upon...
    • The Code 
      Nothing to change or add, leave as is.
    • Optional access restrictions that the Postfix SMTP server applies in the context of the SMTP END-OF-DATA command. See SMTPD_ACCESS_README, section "Delayed evaluation of SMTP access restriction lists" for a discussion of evaluation context and time.
Configure permit_mynetworks

Currently when you send emails form your local network they are relying on the permit_sasl_authenticated to pass the policy stage, not permit_mynetworks because it is not working as expected.

If you are not bothered about this, which most people should not be, just leave these settings as they are and just rely on users being SASL authenticated (permit_sasl_authenticated) which is perfectly acceptable and is probably the best setup.

Why

  • mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128 in main.cf was put there by Ubuntu patches upstream and only allows localhost traffic only.
  • mynetworks_style will not work with the current config because of the presence of mynetworks in main.cf
    • according to the docs: If you specify the mynetworks list by hand, Postfix ignores the mynetworks_style setting.
    • This options has 3 modes, and these are some default examples: 
      # This options has 3 modes, and these are some examples:
host = mynetworks = 127.0.0.1/32 10.0.0.41/32 [::1]/128 [fe80::2a0:98ff:fe24:ff08]/128
subnet = mynetworks = 127.0.0.0/8 10.0.0.0/24 [::1]/128 [fe80::]/64
class = mynetworks = 127.0.0.0/8 10.0.0.0/8 [::1]/128 [fe80::]/64

# Default
Postfix ≥ 3.0: host
Postfix < 3.0: subnet

This default setup does not add your local networks to the mynetworks value so effective it does not work.

Solution

Pick a solution below all of which are slightly different.

  • Virtualmin
    • Webmin --> Servers --> Postfix Mail Server --> General Options --> Local networks (all attached networks): default
      • mynetworks
      • `all attached networks` is an incorrect description and has been reported
    • Webmin --> Servers --> Postfix Mail Server --> General Options --> Automatic local networks
      • mynetworks_style
      • Pick your preferred trusted networks policy.
    • Use the Virtualmin GUI
    • Not available yet
    • I have reported this to the Virtualmin team so this might change. Postfix - `permit_networks` does not work · Issue #2174 · webmin/webmin · GitHub
  • Config Option 1 - Trust connected subnet networks
    • in main.cf delete the line: 
      mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
    • in main.cf add the line: 
      mynetworks_style = subnet
    • Reload the Postfix config.
  • Config Option 2 - Trust connected subnet networks
    • in main.cf delete the line: 
      mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
      • This will restore the default behaviour of Postfix for `mynetworks`by using the `mynetworks_style` setting.
    • Reload the Postfix config.
  • Config Option 3 - Trust the local host and the 10.0.0.0/24 subnet only
    • in main.cf edit the line as follows: 
      mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128

to

mynetworks = 127.0.0.0/8 10.0.0.0/24 [::1]/128 [fe80::]/64
      • As you can see I have added my local network in to what was there. This is equivalent to `mynetworks_style = host"
    • Reload the Postfix config.

Links

General Resource Control
SMTP Server Options
  • smtpd_recipient_limit
    • Webmin --> Servers --> Postfix Mail Server --> SMTP Server Options --> Max number of recipients accepted for delivery: 50
    • Default: 1000
    • This parameter restricts the number of recipients that the SMTP server accepts per message delivery.
  • disable_vrfy_command
    • Webmin --> Servers --> Postfix Mail Server --> SMTP Server Options --> Disable SMTP VRFY command: Yes
    • Default: No
    • This parameter allows you to disable the SMTP VRFY command. This stops some techniques used by spammers to harvest email addresses.
    • SMTP problems : Check if Mailserver answer to VRFY and EXPN requests
      • VRFY and EXPN ask the server for information about an address. They are
        inherently unusable through firewalls, gateways, mail exchangers for part-time hosts, etc.
      • Solution: Disable VRFY and/or EXPN on your Mailserver.
    • Disabling VRFY on InterScan Messaging Security Virtual Appliance (IMSVA) 8.2 | TREND
      • The VRFY clause is a method of verifying the existence of a user on a mail server.
      • You can either verify the existence of particular user or use a wildcard verify (VRFY *) to ask the server to return the complete list of users.
      • On IMSVA version 8.2, VRFY is disabled by default but not on lower versions. The wildcard option (VRFY *) can be exploited by spammers to bulk harvest email addresses so it is necessary that you disable this clause.
    • mail server - Exim - Disable VRFY and EXPN? - Stack Overflow
      • A penetration test has been run on one of my servers that runs Exim for mail and they have this complaint
        • Description: The Mailserver on this host answers to VRFY and/or EXPN requests. VRFY and EXPN ask the server for information about an address. They are inherently unusable through firewalls, gateways, mail exchangers for part-time hosts, etc. OpenVAS suggests that, if you really want to publish this type of information, you use a mechanism that legitimate users actually know about, such as Finger or HTTP.
        • Solution: Disable VRFY and/or EXPN on your Mailserver. For postfix add 'disable_vrfy_command=yes' in 'main.cf'. For Sendmail add the option 'O PrivacyOptions=goaway'.
      • RFC 2505 states:
        • Both SMTP VRFY and EXPN provide means for a potential spammer to test whether the addresses on his list are valid (VRFY) and even get more addresses (EXPN). Therefore, the MTA SHOULD control who is is allowed to issue these commands. This may be "on/off" or it may use access lists similar to those mentioned previously.
        • Note that the "VRFY" command is required according to RFC821
        • To me, this suggests always return a 252, rather than turning if off completely.
      • 2.4. The VRFY and EXPN commands - 49. SMTP processing | exim.org
    • [SOLVED] - postfix: disable answers to VRFY and EXPN requests? | Proxmox Support Forum
      • Hi there,our vulnerability scanner recommends us disabling answering VRFY and EXPN requests by configuringdisable_vrfy_command=yes. Is it safe to do this in a PMG installation?
      • I just did the change, everything looks good so far. I will observe this the next few days and report back.
    • How to disabale VRFY and/or EXPN requests - Support - NethServer Community - Hello, Today, I was doing some security checks on Nethserver using OpenVAS via Ubuntu 18.04. OpenVas find the following vulnerability and suggest me a solution to disable the VRFY and EXPN request on Mailserver. But I don’t know how I can disable these kinds of requests?
  • smtpd_helo_required
SMTP Client Options
  • smtp_use_tls
    • Webmin --> Servers --> Postfix Mail Server --> SMTP Client Options --> Use TLS for SMTP connections?: No
    • Default: No
    • No tooltip
    • Opportunistic mode: use TLS when a remote SMTP server announces STARTTLS support, otherwise send the mail in the clear. Beware: some SMTP servers offer STARTTLS even if it is not configured. With Postfix < 2.3, if the TLS handshake fails, and no other server is available, delivery is deferred and mail stays in the queue. If this is a concern for you, use the smtp_tls_per_site feature instead.
    • This feature is available in Postfix 2.2 and later. With Postfix 2.3 and later use smtp_tls_security_level instead.
    • This option might be obsolete and should note be used. Leaving it on default removes it?
  • smtp_sasl_security_options
    • Webmin --> Servers --> Postfix Mail Server --> SMTP Client Options --> SASL security options: noplaintext, noanonymous 
      noplaintext, noanonymous
    • Default: noplaintext, noanonymous
    • Postfix SMTP client SASL security options; The following security features are defined for the cyrus client SASL implementation.
    • There is a bug with the GUI input field
  • smtp_tls_security_level
    • Before enabling dane-only (Mandatory) you should make sure you server is capable of handling DNSSEC otherwise it will break.
    • dane (Opportunistic) will just fall back to standard email delivery if DNSSEC support is not enabled.
    • DNSSEC should of been enabled in the systemd-resolved (DNS Resolver).
    Webmin --> Servers --> Postfix Mail Server --> SMTP Client Options --> SMTP TLS client security level: Opportunistic DANE TLS
    • Default: Opportunistic DANE TLS
    • No tooltip
    • The default SMTP TLS security level for the Postfix SMTP client.
    • DANE TLS authentication - Postfix TLS Support | postfix.org
      • The Postfix SMTP client supports two TLS security levels based on DANE TLSA (RFC 6698, RFC 7671, RFC 7672) records. The opportunistic "dane" level and the mandatory "dane-only" level.
      • The "dane" level is a stronger form of opportunistic TLS that is resistant to man in the middle and downgrade attacks when the destination domain uses DNSSEC to publish DANE TLSA records for its MX hosts.
        1. If a remote SMTP server has "usable" (see section 3 of RFC 7672) DANE TLSA records, the server connection will be authenticated. When DANE authentication fails, there is no fallback to unauthenticated or plaintext delivery.
        2. If TLSA records are published for a given remote SMTP server (implying TLS support), but are all "unusable" due to unsupported parameters or malformed data, the Postfix SMTP client will use mandatory unauthenticated TLS. .
        3. When no TLSA records are published, the Postfix SMTP client behavior is the same as with may.
      • The "dane-only" level is a form of secure-channel TLS based on the DANE PKI. If "usable" TLSA records are present these are used to authenticate the remote SMTP server.
        1. If "usable" TLSA records are present these are used to authenticate the remote SMTP server. Otherwise, or when server certificate verification fails, delivery via the server in question tempfails.
      • At both security levels, the TLS policy for the destination is obtained via TLSA records validated with DNSSEC. For TLSA policy to be in effect, the destination domain's containing DNS zone must be signed and the Postfix SMTP client's operating system must be configured to send its DNS queries to a recursive DNS nameserver that is able to validate the signed records. Each MX host's DNS zone needs to also be signed, and needs to publish DANE TLSA (see section 3 of RFC 7672) records that specify how that MX host's TLS certificate is to be verified.
SMTP Authentication And Encryption
  • smtpd_sasl_auth_enable
    • Webmin --> Servers --> Postfix Mail Server --> SMTP Authentication And Encryption --> Require SASL SMTP authentication? Yes
    • Default: Yes
    • Enable SASL authentication in the Postfix SMTP server. By default, the Postfix SMTP server does not use authentication.
  • smtpd_tls_auth_only
    • Webmin --> Servers --> Postfix Mail Server --> SMTP Authentication And Encryption --> Disallow SASL authentication over insecure connections?: Yes
    • Default: No
    • When TLS encryption is optional in the Postfix SMTP server, do not announce or accept SASL authentication over unencrypted connections.
  • broken_sasl_auth_clients
    • Webmin --> Servers --> Postfix Mail Server --> SMTP Authentication And Encryption --> Handle non-compliant SMTP clients?: No
    • Default: Yes
    • No Tooltip
    • Enable interoperability with remote SMTP clients that implement an obsolete version of the AUTH command (RFC 4954). Examples of such clients are MicroSoft Outlook Express version 4 and MicroSoft Exchange version 5.0.
  • smtpd_sasl_security_options
    • Webmin --> Servers --> Postfix Mail Server --> SMTP Authentication And Encryption --> SMTP security options --> Reject anonymous logins: ticked
      • Default: ticked
      • This adds the value: noanonymous
    • Webmin --> Servers --> Postfix Mail Server --> SMTP Authentication And Encryption --> SMTP security options --> Reject plain-text logins: unticked
      • Default: unticked
      • This adds the value: noplaintext
    • # These must be in this order
noanonymous, noplaintext
    • No tootip
    • This option needs updating to allow all options even though I will not use them.
    • Postfix SMTP server SASL security options; Restrict what authentication mechanisms the Postfix SMTP server will offer to the client. The list of available authentication mechanisms is system dependent.
    • Warning: it appears that clients try authentication methods in the order as advertised by the server (e.g. PLAIN ANONYMOUS CRAM-MD5) which means that if you disable plaintext passwords, clients will log in anonymously, even when they should be able to use CRAM-MD5. So, if you disable plaintext logins, disable anonymous logins too. Postfix treats anonymous login as no authentication
    • Potential errors if you disable `Plain-text` authentication
      • Remote servers will not be able to send you emails because:
        • It cannot negotiate a suitable connection, this will show up as a fail at the remote server.
        • `Plain-text` is the only mechanims supported by the current setup of Postfix and Cyrus SASL. Plain text is ok because the connection should be encrypted and this is controlled further up by the server (i.e. MTA-STS or enforcing TLS (dane = may thing).
        • When the 2 servers talk they need to negotiate a common Authentication mechanism which both of them have, if not the connection fails.
      • I was not getting this errors when sending from Usermin, but only when receiving emails 
        May 25 10:52:36 server.example.com postfix/anvil[827229]: statistics: max cache size 1 at May 25 10:50:55
May 25 10:52:36 server.example.com postfix/anvil[827229]: statistics: max connection count 1 for (smtp:44.44.44.44) at May 25 10:50:55
May 25 10:52:36 server.example.com postfix/anvil[827229]: statistics: max connection rate 1/60s for (smtp:44.44.44.44) at May 25 10:50:55
May 25 10:50:56 server.example.com postfix/master[817419]: warning: /usr/lib/postfix/sbin/smtpd: bad command startup -- throttling
May 25 10:50:56 server.example.com postfix/master[817419]: warning: process /usr/lib/postfix/sbin/smtpd pid 827226 exit status 1
May 25 10:50:55 server.example.com postfix/smtpd[827226]: fatal: no SASL authentication mechanisms
May 25 10:50:55 server.example.com postfix/smtpd[827226]: warning: xsasl_cyrus_server_get_mechanism_list: no applicable SASL mechanisms
May 25 10:50:55 server.example.com milter-greylist[632]: GeoIP is not available
May 25 10:50:55 server.example.com postfix/smtpd[827226]: connect from server.remotehost.com[44.44.44.44]
      • Postfix SMTP Auth Error "no SASL authentication mechanisms" | The Electric Toolbox Blog - I have been setting up a new mail server recently with Postfix and SMTP Auth, and got the error message "no SASL authentication mechanisms".
      • Postfix - fatal: no SASL authentication mechanisms - Server Fault - Dovecot had been set as the SASL provider but had not been enabled properly, this should not be an issue in Virtualmin.
  • smtpd_tls_security_level
    • Webmin --> Servers --> Postfix Mail Server --> SMTP Authentication And Encryption --> Enable TLS encryption?: If requested by client
    • Default: If requested by client
    • No tooltip
    • The SMTP TLS security level for the Postfix SMTP server.
      • none - TLS will not be used.
      • may (if requested by client) - Opportunistic TLS: announce STARTTLS support to remote SMTP clients, but do not require that clients use TLS encryption. 
      • encrypt - Mandatory TLS encryption: announce STARTTLS support to remote SMTP clients, and require that clients use TLS encryption. According to RFC 2487 this MUST NOT be applied in case of a publicly-referenced SMTP server. Instead, this option should be used only on dedicated servers. 
  • smtp_sasl_auth_enable
  • smtpd_delay_reject
    • Webmin --> Servers --> Postfix Mail Server --> SMTP Authentication And Encryption --> Delay clients with failed logins?: Yes
    • Default: Yes
    • No tooltip
    • Wait until the RCPT TO command before evaluating $smtpd_client_restrictions, $smtpd_helo_restrictions and $smtpd_sender_restrictions, or wait until the ETRN command before evaluating $smtpd_client_restrictions and $smtpd_helo_restrictions.
    • This feature is turned on by default because some clients apparently mis-behave when the Postfix SMTP server rejects commands before RCPT TO.
    • The default setting has one major benefit: it allows Postfix to log recipient address information when rejecting a client name/address or sender address, so that it is possible to find out whose mail is being rejected.
    • This allows the smtp conversation to continue until the point of actually receiving the message before it is rejected, and is useful because it allows full sender and recipient information to be logged. It is also a requirement for helo_restrictions.
Via Config Only

Set these values as specified. Soem of these settings might already be present.

  •  smtp_host_lookup: dns
  • smtp_dns_support_level: dnssec
    • smtp_dns_support_level - Postfix Configuration Parameters | postfix.org
      • Level of DNS support in the Postfix SMTP client.
      • The "dnssec" setting is recommended only if you plan to use the dane or dane-only TLS security level, otherwise enabling DNSSEC support in Postfix offers no additional security. Postfix DNSSEC support relies on an upstream recursive nameserver that validates DNSSEC signatures. Such a DNS server will always filter out forged DNS responses, even when Postfix itself is not configured to use DNSSEC.
      • When using Postfix DANE support the "smtp_host_lookup" parameter should include "dns", as DANE is not applicable to hosts resolved via "native" lookups.
      • As mentioned above, Postfix is not a validating stub resolver; it relies on the system's configured DNSSEC-validating recursive nameserver to perform all DNSSEC validation. Since this nameserver's DNSSEC-validated responses will be fully trusted, it is strongly recommended that the MTA host have a local DNSSEC-validating recursive caching nameserver listening on a loopback address, and be configured to use only this nameserver for all lookups. Otherwise, Postfix may remain subject to man-in-the-middle attacks that forge responses from the recursive nameserver.
        • If you have full DNS hijacking inplace on your network and funnel all of the DNS traffic through your local DNS server as I do with my custom pfSense router, you do not need a DNSSEC-validating recursive nameserver loopback address.
      • Does pfsense just forward the results with the additional flags on it? or do I need to
      • (disable|enabled|dnssec)
      • Default: empty
      • VM Default: dnssec
    • If set to `dnssec`the DNS queries are made to the DNS resolver with these additional flags to make sure DNSSEC validation occurs.
    • Leaving this setting on is the best option, if your updstream resolvers cannot handle DNSSEC validation the flags will just be ignored.
 

Cyrus SASL Authentication Server

  • Remove 'login' as an authentication mechanism as it is old and not used.
  • Edit the config file using the file manager:
    # The config file
/etc/postfix/sasl/smtpd.conf

# Default Contents
pwcheck_method: saslauthd
mech_list: plain login
  • Remove 'login' and save.
  • Go to the dashboard and restart
    • Cyrus SASL Authentication Server
    • Postfix Mail Server

DoveCot (Email)

  • Only allow encrypted connections
    • Webmin --> Servers --> Dovecot IMAP/POP3 Server --> Networking and Protocols --> Accept SSL connections?: Only accept SSL
      • Default: Yes
    • Webmin --> Servers --> Dovecot IMAP/POP3 Server --> SSL Configuration --> Disallow plaintext authentication in non-SSL mode?: Yes
      • Default: No
  • Strong Encryption
    • Webmin --> Servers --> Dovecot IMAP/POP3 Server --> User and Login Options --> Authentication Methods: Plain-text
    • Default: Plain-text, login
    • Authentication — Dovecot documentation
    • Authentication (SASL) Mechanisms — Dovecot documentation
    • If `login` is deselected it will be removed from the list and will not reappear unless you manaully add this option into the config file.
    • If you select `Cram-MD5`only, because this option requries local hashing of passwords it needs an intermediary database for something so you will get the following error if this database has not been setup: 
      May 24 17:36:53 server.example.com dovecot[536102]: imap-login: Error: auth-client: conn unix:login (pid=1125,uid=0): Timeout waiting for handshake from auth server. my pid=693884, input bytes=0
May 24 17:36:12 server.example.com dovecot[1125]: master: Error: service(auth): command startup failed, throttling for 60.000 secs
May 24 17:36:12 server.example.com dovecot[536102]: auth: Fatal: CRAM-MD5 mechanism can't be supported with given passdbs
  • Save email with CRLF line endings?  / Windows new line support?
    • Webmin --> Servers --> Dovecot IMAP/POP3 Server --> Mail Files --> Save email with CRLF line endings? Yes
    • Default: Default (No)
    • I have not tried this to see what real difference it makes and while is Dovecot making changes to the file.
    • mail_save_crlf - Dovecot Core Settings — Dovecot documentation
    • Enabling this makes saving messages less CPU-intensive, especially with the sendfile() system call used in Linux and FreeBSD. However, enabling comes at the cost of slightly increased disk I/O, which could decrease the speed in some deployments.

ClamAV (SPAM)

nothing to change

SpamAssassin (SPAM)

  • Earlier on we choose the SPAM filtering option that allowed per-domain filtering and these settings blow will reflect that.
  • The options you choose here depend and how you want the SPAM to be controlled, by you or by the individual virtual server owners.
  • I prefer to control the SPAM at the server level as people don't want SPAM and I would only need to control it on one place however it is useful for clients to be able to alter their particular setup.
Basic Settings
  • Set maximum message size to process (if not already)
    • Virtualmin Global: Virtualmin --> Email Settings --> Spam and Virus Scanning --> Maximum message size to process: unlimited
    • Antivirus configuration - #3 by ID10T - Virtualmin - Virtualmin Community
      • I had a problem with spam completely bypassing filtering. It turns out that 500KB size limit was coming into play. From looking at the configuration page it isn’t 100% clear to me if spam and virus filtering are both affected by the single setting.
    • You can check the config file /etc/spamassassin/local.cf and the following will be commented out for unlimited email size.
      • body_part_scan_size
      • rawbody_part_scan_size
  • Allow DNS lookups
    • Webmin --> Servers --> SpamAssassin Mail Filter --> Miscellaneous User Options --> Can SpamAssassin do DNS lookups? Yes
    • (dns_available)
    • Default: Default (Test)
    • dns_available - Mail::SpamAssassin::Conf - SpamAssassin configuration file
    • Since version 3.4.0 of SpamAssassin a default setting for option dns_available is yes. A default in older versions was test. I will remove this line when Virtualmin is updated to reflect this.
Filters Configuration

We will configure some SpamAssassin rules. These can change from setup to setup and why they have their own section. We will be using "per domain" settings as this is the best setup.

  • Class emails as SPAM if they fail SPF check
    • Webmin Global: Webmin --> Servers --> SpamAssassin Mail Filter --> Header and Body Tests --> Switch to advanced mode --> SpamAssassin test scores: SPF_FAIL = 10.00
      • Any email that fails SPF checks should just be failed and then deleted at the server level.
    • Virtual Server: Virtualmin --> Mail Options --> SpamAssassin Configuration --> Header and Body Tests --> Switch to advanced mode --> SpamAssassin test scores: SPF_FAIL = 10.00
      • The availability of this setting is only available if you selected standalone mode.
  • Automatically delete SPAM / Automatically delete Virus
    • Webmin Global (only when using spamc):
      • Webmin --> Servers --> SpamAssassin Mail Filter --> Procmail Spam Deilvery --> Action for messages classified as spam: Throw away
      • This will delete emails that fail the 'header and Body Tests' that are run by 'Webmin Global' only as the per-domain tests have not been run yet, so effectively this will only currently delete any emails that fail SPF tests which is what I want done at the server level.
      • there is no option to delete email virus at the webmin level because there is no way to configure it through the GUI. Probably can be done manually. Cannot delete emails with spam at the server level, on at virtual server level · Issue #818 · virtualmin/virtualmin-gpl · GitHub
      • This option will be getting hidden when Virtualmin is set top filte on a per domain basisso will not be usable in future versions unless you are set to filter SPAM globalluy only ising spamc.
    • Virtualmin Defaults (only when using per domain filtering):
    • Virtual Server (only when using per domain filtering):
      • Virtualmin --> Mail Options --> Spam and Virus Delivery --> Destination for spam emails: Throw away
      • Virtualmin --> Mail Options --> Spam and Virus Delivery --> Destination for virus emails: Throw away
      • Virtualmin --> Mail Options --> SpamAssassin Configuration --> Procmail Spam Deilvery --> Action for messages classified as spam: Throw away
    • Usermin (only when using per domain filtering):
      • Usermin --> Mail --> SpamAssassin Mail Filter --> option not present, might be permissions
DNSBL (a.k.a. RBL)

nothing here to change. SpamAssassin has inbuilt DNSBL that are pre-configured.

SpamAssassin Addons

These can be used to extend SpamAssassin's features but are not required.

Razor Spam Detector (SPAM / SpamAssassin)

I need to install this and use it before adding settings here.

Pyzor (SPAM / SpamAssassin)

I need to install this and use it before adding settings here.

DCC (SPAM / SpamAssassin)

I need to install this and use it before adding settings here.

SPF (Email)

SPF: HELO does not publish an SPF Record (SPF_HELO_NONE)

This SPF failure does not add many points onto your SPAM score but with it you will not get 100%. In the future this failure could have more of an impact and because it is so easy to fix it, you should.

The Error

SPF_HELO_NONE        SPF: HELO does not publish an SPF Record

Testing

Cause

Your server does not have a SPF record in it's DNS Zone. Your server's hostname is use in the sending and receiving of email and this is a test that is done to check the server is valid.

Solutions

  • Via Virtualmin (preferred method)
    • This assumes you have your server's virtual server visible
      • Virtualmin --> System Settings --> Virtualmin Configuration --> Configuration category: SSL Settings --> Setup Let's Encrypt SSL certificate for hostname
    • Virtualmin --> DNS Settings --> DNS Options
      • SPF record enabled: Yes
      • Action for other senders: Disallow
      • Allowed sender hostnames: Remove the hostname/domain
      • Save
    • The record you have created will work but currently the SPF builder is not very controllable. You should edit the record your record to make it look like Standard SPF Record (Improved) shown below when Virtualmin improves this page or you can use the Webmin option below because the mx entry will cause a SPF failure as there should be no mail server on the server's hostname thus no mx entry ever.
      • Virtualmin --> DNS Settings --> DNS Record --> Manually Edit Records
  • Via Webmin
    • Webmin --> Servers --> BIND DNS Server --> Existing DNS Zones --> your server hostname (eg server.example.com) --> Edit Zone Records File
    • Add a suitable SPF record, the IP and domain should be your server's IP and hostname. You can always copy a SPF record from one of your live domains. 
      # Standard SPF Record
server.example.com.	IN	TXT	"v=spf1 a mx a:server.example.com ip4:31.31.31.31 -all"

# Standard SPF Record (Improved)
server.example.com.	IN	TXT	"v=spf1 ip4:31.31.31.31 -all"
    • NB:
      • Ignore the warning about this being controlled by Virtualmin.
      • Be careful with what you alter here.
      • Make a backup up of the zone before you do anything.
      • This record potentially could get removed with updates in the future.
      • If your local IP is present in the SPF record you should delete this as it is not needed and can be a security risk. 
        ip4:10.0.0.23

DKIM (Email)

Greylisting (Email / Postgrey) (optional)

Even if you do not use this feature, you should still purge the lists so you don't have to later.

  • Virtualmin --> Email Settings --> Email Greylisting --> Enable Greylisting
  • Purge the default `Whitelisted Clients` and `Whitelisted recipients` as these are old and insecure
  • Information
    • this will delay email getting delivered to mailboxes and is usually about a 5 minutes delay.
    • Greylist is a technique to reduce spam by initially rejecting email the first time another mail server tries to contact your server. Real mail servers will re-try after a short delay, but those operated by spammers typically will not. Thus legitimate email still gets delivered, but spam does not.
    • In addition, whitelists for SMTP servers and email recipents can be managed.
    • This uses the Postgrey package.
    • Greylisting can cause a delay to emails getting delivered to your mailbox, becasue this is how it works, it waits for the remote server to re-send the email to make sure this email is legit.
    • The default Postfix settings will usually allows a retry every 5 minutes.
    • The resend timer on Postfix is controlled here: Webmin --> Servers --> Postfix Mail Server --> Delivery rates

Virtualmin (Email)

  • Mail Rate Limiting
    • Virtualmin --> Email Settings --> Mail Rate Limiting --> Rate limiting enabled?: Yes
    • Virtualmin --> Email Settings --> Mail Rate Limiting --> Global message limit:  50 per hour
      • This will apply per virtual server and is not one total value for the whole server.
    • This prevents you server from spamming the world if a domain becomes compromised.
    • You can override this for particular domains if they have greater need without risking the rest of the server.
    • I think this uses the Greylist MFilter server to handle the rates.
  • Mail Client Configuration (optional)
    • Virtualmin --> Email Rate Limiting --> Mail Client Configuration --> Enable mail client autoconfiguration?: Yes
    • This option will create an Autoconfiguration file for email clients in the location of: 
      http(s)://example.com/mail/config-v1.1.xml
    • The information to build the XML is pulled from
      • Virtualmin --> System Settings --> Server Templates --> yourtemplate --> Edit tempalte section: Mail client auto-configuration
    • Links

System and Server Status / System Monitors

Now you have set up your services you should make sure they are monitored

You can setup other monitors for services and other operations of your server making "System and Server Status" very powerful.

Any monitored services will be display in the virtualmin dashboard widget called "System Monitors"

Monitors can be configured as follows:

  • Show the Server Monitors on the dashboard
    • Webmin --> Tools --> System and Server Status --> Settings Cog --> Show monitors on Dashboard: Yes
  • Enable required monitors
  • Watchdog (almost)
    • Webmin --> Tools --> System and Server Status --> Edit monitor --> If monitor goes down, run command:
    • You can configure each monitor to run a custom command if a monitor has failed, but this is not the same as just checking a tickbox to restart the service if not running.

Notes

  • System and Server Status | Webmin - About This page covers the use of Webmin’s System and Server Status module, which can be used to check for and report down systems, failed servers, network outages and other problems.The module This module allows you to monitor the status of various servers and daemons running on your system, so that you can easily see which are running properly and which are down. It can also be configured to check the status of servers on a regular schedule, and to email you or run a command if something goes down.

System Notifications (Email)

Configure Webmin System Email

  • What is the Webmin's system email address
    • Webmin --> Webmin Configuration --> Sending Email --> Send Email
    • The email will should be in the format similiar to webmin@server.example.com
  • Configure an alternative email address (optional)
    • Setup your alternative email address and have the details to hand.
    • if your email address is located on the same server
      • Webmin --> Webmin Configuration --> Sending Email --> Send email using: Local mail server command
      • The rest of the settings will now be of no use as the emails are routed internally before being sent
    • If your email is not on the same server
      • Webmin --> Webmin Configuration --> Sending Email --> Send email using: Via SMTP to remote mail server
      • Configure the rest of the settings as required
      • Keep the settings as secure as posible.
  • Send a test email from Webmin
    • Webmin --> Webmin Configuration --> Sending Email --> Send Email
      • Make sure you have this email address whitelisted to gurantee delivery

Notifications

It is important to be notified when there are issues on your server. Some of these might of been addressed elsewhere in this tutorial this section is here to help me make sure I have not missed anything.

Configure these to match your needs

eg:

  • System Monitors
    • Webmin --> Tools --> System and Server Status
    • Ypu can add cusomt monitors here that will send an email to the system email.
  • Bandwidth monitoring
    • Virtualmin --> System Settings --> Bandwith Monitoring
  • Disk Quota Monitoring (Pro)
    • Virtualmin --> Limits and Validation --> Disk Quota Monitoring
  • Webmin errors
  • Security updates
    • Webmin --> System --> Software Package Updates --> Scheduled Upgrades
    • This is disussed later on in the updates section
  • Webmin Actions Log
    • Webmin --> Webmin --> Webmin Actions Log --> Email notifiction
    • All actions of Webmin can be monitored and an email sent if triggered.
  • Postfix
    • Webmin --> Server --> Postfix Mail Server --> General Options --> Most Useful General Options --> What trouble to report to the postmaster:
  • Lets Encrypt

DMARC (Email)

Theme

These are all per user as there are no global theme defaults except for a couple of options (Login page color palette, Forbid access to theme config for users).

  • Set Dark Mode
    • Theme Configuration --> Configuration category: General defaults --> Login page color palette: Dark
    • Click on the Day/Night button to enable dark mode.
    • Each user has to choose to use Dark mode by clickling on the 'Day/night mode toggle' button
  • Make icons coloured
    • Theme Configuration --> Configuration category:  Table display --> Show table icon links in gray scale unless hovered: No
  • Add animations on hover
    • Theme Configuration --> Configuration category:  Table display --> Show on-hover animation for table icon links: Yes
  • Prevent users changing the theme
    • Theme Configuration --> Configuration category: General defaults --> Forbid access to theme config for users: Yes
    • Theme Configuration --> Configuration category: Navigation menu --> Show Day/Night mode button: No
      • this option is not pushed to all users
  • Add 'Administrator' tag to the menu
    • Useful for knowing you are logged in with an admin account.
    • Theme Configuration --> Configuration category: Navigation menu --> Show HTML snippet: <code>Administrator</code>
    • Theme Configuration --> Configuration category: Navigation menu --> Show HTML snippet for administrators only: Yes
  • Add a seperator between the "Virtual Server" and "Global" options (optional)
    If the code below does not work it might be a line ending issue or tabs converted to spaces, but easily fixed.

    • Add the code as follows
      • Theme Configuration --> Theme Extensions Edit extension file: /etc/webmin/authentic-theme/styles.css 
        /** Split Virtual Server and Global menu sections - v1.0 **/

/* Default/Day Mode - Section Separator */
#customSectionSeparator {
	width: 50%;
	margin-left: 25%;
	border-top: 2px solid #f5f0fffa !important;
	margin-top: 30px !important;
	padding-top: 30px !important;
}

/* Dark Mode - Section Separator */
html[data-theme="gunmetal"] #customSectionSeparator {
	border-top: 2px solid #00000054 !important;
}

/* Default/Day Mode - Search Box - Full border */
#webmin_search_form > div.form-group .form-control.sidebar-search {
	border: 1px solid #ffffff29 !important;
}

/* Dark Mode - Search Box - Full border */
html[data-theme="gunmetal"] #customSectionSeparator {
	border-top: 2px solid #ffffff1f !important;
}
      • Theme Configuration --> Theme Extensions Edit extension file: /etc/webmin/authentic-theme/scripts.js 
        /** Split Virtual Server and Global menu sections - v1.0 **/

// Add in a div to allow correct sizing of Section Separator
function addCustomSectionSeparator()
{
	// Only add the Section Seperator if it does not exist and the Virtualmin menu is present
	if (document.getElementById("customSectionSeparator") == null && $(document.getElementsByName("dom")).is(":visible"))
	{
		// Get the container
		var container = document.getElementById("webmin_search_form").parentElement;

		// Build the code
		var myCreatedElement = document.createElement("div");
		myCreatedElement.setAttribute("id","customSectionSeparator");

		// Insert the code
		container.insertBefore(myCreatedElement, container.firstChild);        
	}
};

// Add Section Separator on initial page load (remember Virtualmin is a single page system)
$(document).ready(function()
{
	addCustomSectionSeparator();
});

// Add Section Separator on page changes (remember Virtualmin is a single page system)
$(document).change(function()
{	
	setTimeout(addCustomSectionSeparator, 500);
});

Security

  • Some of these might require the editing of Account plans or server templates. I might move them if required.
  • Only if you knoiw what you are doing, you can use ConfigServer Security & Firewall (csf + lfd) instead of FirewallD and Fail2Ban.
  • Enable FirewallD
    • Webmin --> Networking --> FirewallD
  • Enable Fail2Ban
    • Webmin --> Networking --> Fail2Ban
  • Force HTTP to HTTPS (optional)
    • Virtualmin --> Web Configuration --> Website Options --> Redirect all requests to SSL site: Yes
    • This is normally done with your CMS or by you in .htaccess.
    • What this does
      • This creates an Apache Directive to perfom the redirect as follows in /etc/apache2/sites-available/example.com.conf: 
        RewriteCond %{HTTPS} off
RewriteRule ^/(?!.well-known)(.*)$ https://%{HTTP_HOST}/$1 [R]
      • This appears as a redirect in: Virtualmin --> Web Configuration --> Website Redirects
      • How to manage URL redirects – Virtualmin - This tutorial will cover how to setup URL redirects. A URL redirect allows you to make one URL redirect to another of your choice.
  • Force FTP to only use FTPS/TLS.
    • This is not yet available as a GUI option
    • Edit the config file: Webmin -->Servers --> ProFTPD Server --> Edit Config Files --> Editing config file: /etc/proftpd/conf.d/virtualmin.conf:
    • Enforce TLS  by changing: 
      TLSRequired off --> TLSRequired on
    • Save the config.
    • Apply the changes (this will restart the ProFTPD service).
  • Disable SSH access from users
    • The `SSH Login` is not enabled by default in `Administrator's Webmin modules` and thus should not be on.
    • You can check here: Virtualmin --> Manage Virtual Server --> Edit Owner Limits --> Other restrictions --> Allowed login type
  • Remove terminal from users
    • The `Terminal` is not enabled by default in `Administrator's Webmin modules` and thus should not be on.
    • Currently there is no way to changes this after create the Virtual Server.
    • Disable the option in the relevant server template, or the default server template.
  • Disable 'Syncing your SQL and hosting account' (optional)
    • If someone comprimises your CMS they can get your account usename and password.
    • This is only dangerous if you use the credentials in a web application where the details could be retrieved.
  • Disable Webmin root account.
  • Remove the root account from SSH.
  • Disable Usermin (optional)
  • Restrict access to Webmin by IP or Hostnames.
    • Webmin --> Webmin Configuration --> IP Access Control --> Allowed IP addresses:
  • Restrict access to Usermin by IP or Hostnames (or Disable Usermin).
    • Webmin --> Usermin Configuration --> IP Access Control --> Allowed IP addresses:
  • Increase password strength requirements
    • Virtualmin --> System Settings --> Virtualmin Configuration --> Configuration category: Defaults for new domains --> Length of randomly generated password: 20
    • Virtualmin --> System Settings --> Virtualmin Configuration --> Configuration category: Defaults for new domains --> Characters for random passwords: 
      !"#$%&()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{|}~
    • Are these pointless becasue I use Virtualmin
      • Webmin --> System --> Users and Groups --> Settings Cog --> Configuration category: Password restrictions --> Minimum password length: 20
      • Webmin --> System --> Users and Groups --> Settings Cog --> Configuration category: Password restrictions --> Perl regexp to check password against:
      • Webmin --> System --> Users and Groups --> Settings Cog --> Configuration category: Password restrictions --> Prevent passwords containing username?: Yes
  • Automatically generate passwords
    • Virtualmin --> System Settings --> Virtualmin Configuration --> Configuration category: Defaults for new domains --> Password field type: Randomly generated password
    • This does not directly make things more secure but it makes things easier by generating a suitable password as specified above.
    • This saves you having to a click one button.
  • Webmin Authentication
    • We need to set some limits for authentication to prevent brute force attacks and other misuse.
    • Webmin --> Webmin configuration --> Authentication --> Password timeouts: Enabled
      • When Enable password timeouts is selected, Webmin will detect multiple failed login attempts from the same IP address and lock that host out for a configurable amount of time. This feature should always be turned on, as it stops attackers using millions of login attempts to guess passwords on your system.
      • Password timeouts and expiry - Need these options clarifying - Webmin - Virtualmin Community
        • I’ll add a tooltip with details on what password timeouts mean, but basically when this is enabled there will be an increasing delay between failed login attempts.
        • There’s no way to configured the password timeout delays … they are fixed in Webmin
    • Webmin --> Webmin configuration --> Authentication --> Failed login blocks:
      • Block hosts with more than 5 failed logins for 3600 seconds.
        • hosts in this context means IP addresses
      • Block users with more than 5 failed logins for 3600 seconds.
        • This is referring to Webmin users and not Unix user acconts.
      • Also lock users with failed logins: unticked
        • Locked accounts will not become active again withou manual intervention so this shoul donly be enabled when required.
    • Webmin --> Webmin configuration --> Authentication --> Authentication Options:
      • Auto-logout after: 60 minutes of inactivity
        • If you have client's it is best to enable this option, if there is just you and your admin is not exposed on the internet you can ignore this one, but if not sure definately add it.
        • A good fall back is 240 Mins which prevents mistakes if there is just you at home or the office.
      • Offer to remember login permanently?
        • If you have client's it is best to disable this option, if it is just you it can be left enabled
        • When selected, the cookie sent to the user’s browser will be marked to indicate that it should be saved even if the browser is shut down and re-run later.

Further Settings

If you are unsure about any settings, do not change them.

File Manager

Currently these settings on a per-user basis

  • File Manager Configuration --> Configuration category: Advanced options --> Hide column containing action icons: Yes --> No

Virtualmin

  • Put your holding page files in your skeleton directory /etc/skel
  • Virtualmin --> Virtualmin Configuration --> Configuration category: SSL settings --> Show Let's Encrypt error at domain creation time?
    • This will notify you of any errors and can be very useful.
  • Configure the Columns to show on 'List Virtual Servers' page
    • Virtualmin --> Virtualmin Configuration --> Configuration category: User interface settings --> (Columns to show && Feature columns to show)
    • Configure as shown below for a good start.
  • Go through the unused modules and add any that you need.
  • Go through the rest of the Virtualmin --> Virtualmin Configuration

Webmin

  • Configure the System Time (System Clock)
    • Install the required binaries using the terminal
      apt-get install ntpdate
      Otherwise you will get this error 
      NTP time synchronization failed : Missing ntpdate and sntp commands
    • Webmin --> Hardware --> System Time --> Change timezone --> Change timezone to: your local timezone
    • Webmin --> Hardware --> System Time --> Time server sync

      • Configure the settings as shown above.
      • You can set your own preferred NTP server if you want.
      • Set hardware time too
        • This is fine for Virtual machines (i.e. KVM Guests) because KVM provides guest virtual machines with a paravirtualized clock (kvm-clock).
      • The minutes and hours are initially randomly selected and you can use those times if you want.
      • NB: To de-select or multi-select, use the Ctrl button while clicking.
  • Go through the rest of the Webmin --> Webmin Configuration
    • You should not need to touch anything here
  • Enable Bandwidth Monitor (optional)
    • Webmin --> Networking --> Bandwidth Monitoring
    • Bandwidth Monitoring | Webmin - The Bandwidth Monitoring module can be used to create simple reports on bandwidth usage by port, host, protocol and time for traffic sent from or routed through your system. It is useful for both stand-alone hosts, and those that act as a gateway (possibly with NAT) for a network. Before it can be used, the module must setup several firewall rules and a syslog entry to capture traffic sent and received via your system.
    • This is not required for 'Bandwidth Quotas'
    • Useful to track how active websites are over time and this will show you traffic per ports.

Usermin

  • This needs to be done if you are going to allow clients to login.
  • I will not give clients access so I have not done this section.
  • Configure available modules for Usermin
    • Webmin --> Usermin Configuration --> Available Modules

WAF / Firewalls / IDS / IPS

You should consider if the default FirewallD + fail2ban does what you want. If you don't know just leave the settings as they are.

Jails

I don't know how to configure these or how to use them, this section is a placeholder if this changes.

Virtualmin Pro

There might be some additional Pro only options that need configuring. I will update this section when I start using Pro.

Reseller

  • When you have pro there is an additional category in Virtual configuration
    • System Settings --> Virtualmin Configuration --> Configuration category: Reseller settings
  • Enable Terminal for a Reseller
    • Terminal - xterm available on reseller - Virtualmin - Virtualmin Community
      • Information
        • A reseller account doesn’t have a real system user, thus can’t use the terminal (which is a system login). It’s probably possible to create a system user with the same name, though, and grant them access to the Terminal in Webmin Users.
        • But, you’re probably expecting them to have access they won’t have. The terminal would have user-level permissions, and the domains the reseller manages will not be owned by or in the same group as the reseller, and so the reseller won’t be able to do anything with their domains content. You’d need to…do something else. I’m not sure what the right option would be. Maybe add all the domain groups as secondary groups for the reseller. As long as all the domains files are group writable, that’d work, but also has some potential risks. Linux has finer-grained access control in a variety of ways, which might also be an option
      • Solutions described (I have not verified these)
        • System Settings --> Virtualmin Configuration --> Configuration category: Reseller --> Additional modules for resellers: Add Terminal 
        • System Settings --> Virtualmin Configuration --> Configuration category: Reseller --> Create Unix user for new resellers: Yes
        • Webmin --> Webmin Users --> reseller --> bypass the warning --> Available Webmin modules --> Tools: tick Terminal
      • Bug: In Authentic theme, the Terminal icon will show but the Terminal menu will not show

Set your NAT Static IP / Finalise your IP address

  • If you are using Virtualmin behind a NAT, you should set your permanent local Static IP now.
  • You can always easily update DNS records later but why do things twice. In fact Virtualmin will notice the change ans ask you to make the change.
  • When you change from DHCP make sure you set your Gateway and DNS servers on the network card, via Webmin otherwise you will get connectivity issues that will difference from the internal and external networks to your server aswell as Virtualmin will not be able to perform outward connections. When you are on DHCP you are supplied with the Gateway and DNS server settings automatically.
    • Webmin --> Networking --> Network Configuration --> Routing and Gateways --> Default router --> Gateway: 10.0.0.1
    • Webmin --> Networking --> Network Configuration --> Hostname and DNS Client --> DNS servers: as per the systemd-resolved (DNS Resolver) section:
      • 10.0.0.1
      • 127.0.0.53, then 10.0.0.1
      • 9.9.9.9 or 8.8.8.8 etc.. (if not DNS hijacking and/or just using external DNS)
  • When you reload Webmin after changing IP address you will be presented with this message and you should use the link to changes the IPs as advised.
    • The controlpanel host address was not changed with this tool.
      • Webmin --> Networking --> Network Configuraiton --> Host Addresses --> <10.0.0.137/server.example.com , web> --> IP Address:
      • Still showing 10.0.0.253, so i changed it to --> 10.0.0.44
  • Configure your firewall to forward port as required (and using any security polices you have in place).

Create your Primary Hosting account (example.com)

Account Creation

  • Create a Virtual Server with your primary domain
    • Virtualmin --> Create Virtual Server
    • Use the domain selected by you in the preperation stage.
    • With an email dedicated for the system to use to send notifications, e.g. no-reply@example.com.
      • On some configurations a dedicated email address might be required, but not always.

User Settings

  • Remove SSH from the account owner
    • Virtualmin --> Manage Virtual Server --> Edit Owner Limits --> Other Restrictions --> Allowed login type: Email and FTP
  • Disable any email on the primary account (optional)
  • Add an email addresse(s) to the domain
    • Virtualmin --> Edit Users --> Add a website FTP access user
    • Configure the email address and password.
    • Other user permissions --> Login permission: Email only
      • This ensures this user is just an email account.
    • Set any other option that you need, but this is enough.

Domain Settings

  • Create nameservers (NS entries)
    • Virtualmin currently does not allow you to use nameservers that do not already exist and you cannot automatically create custom nameservers.
    • DNS Frequently Asked Questions – Virtualmin
      • Virtualmin --> DNS settings -- DNS Records --> Create Record of Type: NS Name Server
        • Record name: same as domain
        • Record type: NS - Name Server
        • Cache time: Domain default
        • Name server DNS name: ns1
        • Record Comment: leave blank
        • Repeat for ns2
      • Virtualmin --> DNS settings -- DNS Records --> Create Record of Type: A - IPv4 Address
        • Record name: same as domain
        • Record type: A - IPv4 address
        • Cache time: Domain default
        • IPv4 address: your external ip
        • Record Comment: leave blank
        • Repeat for ns2
  • SSL Certificate (Lets Encrypt)
    • Virtualmin --> Manage Virtual Server --> Setup SSL Certificate
    • If your domain is not pointing to your Virtualmin server, then a Lets Encrypt Certificate will not be requested and it will have to be done manually later.
    • How to add an SSL certificate – Virtualmin - These instructions will tell you what fields to fill in, if not already.
  • Set the Default website for IP address / Shared SSL / Default Domain
    • Virtualmin --> Web Configuration --> Website Options --> Default website for IP address: Yes
    • Tooltip: When this field is set to Yes, this virtual server's website will be served by Apache when it receives a request that doesn't match any other virtual server on the system. This typically happens if a user uses a URL with an IP address in it, or a hostname that resolves to your system but does not match any Virtualmin domain.
    • You might see one of the following variants of the option depending which domain is selected as the default website:
       
  • Configure DNSSEC (optional)
    • This is an important technology and prevents your domain from getting spoofed.
    • DNSSEC should of been enabled by your 'Internal' template, but if not, go here and enable it.
      • Virtualmin --> DNS Options --> DNSSEC signature enabled: Yes
      • Click 'Save'
    • Automatic Key Re-Signing
      • Webmin --> Servers --> BIND DNS Server --> DNSSEC Key Re-Signing --> Automatic key re-signing enabled?: Yes
      • If you do not enable this, your DNSSEC keys will expire and give you one or both of these errors (The following zones have expired DNSSEC signatures):ggggg

    • Goto Virtualmin --> DNS Options --> DNSSEC zone keys --> DS records for registrar
      • You will see something similiar to the image below. I have annotated the image as it was tricky figuring out which number did what.
    • Now you have the relevant information, you need to enter it at your registrar

      • In this example you can see the numbers in parenthesis which is the same as the numbers in your DS records for registrar.
      • Your registrar will probably have a form similiar to this as long as they support DNSSEC, not all do.
      • DNSSEC Guide — BIND 9 9.19.19-dev documentation
        • This document provides introductory information on how DNSSEC works, how to configure BIND 9 to support some common DNSSEC features, and some basic troubleshooting tips.
        • This is well written and easy to read. I found it very helpful.
  • Enable TLSA records
    • Virtualmin --> DNS Settings --> DNS Options --> TLSA records enabled: Yes
    • Currently you cannot set this as an option in the Server Template so it has to be enabled manually.
  • Configure the SPF 'Fail Qualifier'
    • The server template does not handle SPF records creation properly so we have to set the 'Fail Qualifier' manually.
    • Virtualmin --> DNS Settings --> DNS Options --> Action for other senders: Disallow
      • <default> = none
      • Disallow = -all
      • Discourage = ~all
      • Neutral = ?all
      • Allow = all
  • Add Website Aliases
    • Do you have several versions of the domain names (eg example.com, example.co.uk, example.uk) that you want to point to the same website? You can add them now.
    • Virtualmin --> Create Virtual Server --> Alias of example.com

Install Serverwide Apps

Create One location for the Apps

Now you have created your primary hosting account I would install your single copy of phpMyAdmin. further details instructions can be found in the 'Serverwide Apps' section above, however the instructions below will work for everyone.

  • Create a directory ~/public_html/apps/ on your primary domain which will look like https://www.example.com/apps/
  • Restrict access to the /apps/ folder
    • I recommmend this restricting access.
    • It would be better if Apps were hidden away like cPanel. Edit as required.
    • create a .htaccess using the content below. 
      # DISABLE DIRECTORY INDEXES
Options -Indexes

# RESTRICT ACCESS TO DIRECTORY BY IP ADDRESS
# Include in .htaccess of any directory
<RequireAny>
    Require all denied
    #Require ip 1.2.3.4
    #Require ip 5.6.7.8/12
    
    # If local server access to the directory is required
    # add the following; include the server IP addresses (IPv4 & IPv6)
    Require local
    #Require ip 192.168.1.0/24
    #Require ip 2001:0db8:85a3:0000:0000:8a2e:0370:7334
</RequireAny>

Install phpMyAdmin Centrally

  • Create a separate database with its own user
  • Install using the Virtualmin Install Script, using the database you just created, to following location www.example.com/apps/phpmyadmin/
  • Change the database user if required.
  • Adding a Virtualmin Dashboard Menu Item using the GPL Theme based solution
    • Theme Configuration --> Configuration category: Navigation menu --> Extra entries 
      {
    "extra": [{
        "title": "phpMyAdmin",
        "link": "https://www.example.com/apps/phpmyadmin/",
        "icon": "php",
        "level": "0",
        "target": "_blank"
    }]
}
      • This menu item will be visible for administrators only, but can be changed.
      • Don't forget to change the link to match your primary domain name or other target.
      • or further information and methods see the Custom Menu Links section.
      • Currently the field will not except this format and needs to be flattened. 
        {"extra":[{"title":"phpMyAdmin","link":"https://www.example.com/apps/phpmyadmin/","icon":"php","level":"0","target":"_blank"}]}

Other Centralised Apps

If you have any other apps that shoul dbe centralised follow the procedure outlined above.

  • Roundcube
    • I am not sure if Usermin is better instead of Roundcude
    • the Usermin dynamic from webmail.example.com can be changed to the rouncube directory

Branding (optional)

Styling your Virtualmin installation can be useful for identifying your dev and live sites

  • Add a logo to the login page
    • Theme Configuration --> Theme Logos
  • Style the Theme Background
    • Theme Configuration --> Theme Backgrounds
  • Show real hostname instead of name from URL? (optional)
    • Webmin --> Webmin Configuration --> Authentication --> Show real hostname instead of name from URL?
    • This is useful for identifying live and dev servers when you are not using the system hostname to login for admin purposes.
  • Webmin --> Webmin Configuration --> Authentication --> Pre-login banner
    • I have not used this

Final Things

  • Enable a real SSL certificate from Let's Encrypt for your Virtualmin hostname (eg server.example.com)
    • Virtualmin --> System Settings --> Virtualmin Configuration --> Configuration category: SSL settings --> Setup Let's Encrypt SSL certificate for hostname: Yes
    • Virtualmin --> System Settings --> Re-Check Configuration (this is done below so is not needed here at this time)
  • Get Virtualmin to check various settings and configurations. This also runs some housekeeping tasks.
    • Virtualmin --> System Settings --> Re-Check Configuration
  • Check Webmin can "still" send emails
    • Webmin --> Webmin Configuration --> Sending Email --> Send Email

etckeeper

I am using Ubuntu LTS 22.04 and etckeeper is installed, you need to check if it is installed on your OS automatically, if not you should consider doing it manually.

  • Virtualmin installs this by default on systems that have a etckeeper package availavble that can set itself up automatically.
  • This is not a substitute for backups, but it does allow you to see exactly what changes you made, which might help you fix things if you make a mistake that breaks something and you don’t remember what you changed to get there.
  • etckeeper allows the contents of /etc to be stored in a Version Control System (VCS) repository. It integrates with APT and automatically commits changes to /etc when packages are installed or upgraded.
  • The location for the GIT repo in Virtualmin is: /etc/.git/
  • etckeeper also sets up a daily cron job.
  • The changes in /etc files are stored in the GIT repository that was created, forever.
  • backup - include etckeepr · Issue #2238 · webmin/webmin · GitHub
    • The backups for etckeeper must be handled by etckeeper module, whenever we create it.
    • For now, the best solution is just to include it in the Webmin backup manually.
      • Webmin --> Backup Backup Configuration Files --> Backup now --> Include in backup --> Other listed files .. 
        /etc/.git
  • add an `etckeeper` module · Issue #2240 · webmin/webmin · GitHub

Backup Strategy (Policy)

  • Configure a backup strategy.
    • to be added later
  • how do i backup the whole server
  • how do i backup webmin and virtualmin settings. are these the same as the /etc/
  • Back these up
    • Webmin Config (modules + etc) + etckeeper (optional)
    • Virtualmin config
      • how can I backup virtulmin configuration without backing up a virtual server
      • there should be an option to backup the virtulamin config settings only. this could be represeneted by the hostname of the server web.svchost.uk
    • Virtualmin Virtual Servers
    • Encryption Keys (if these are separate)
  • Offsite backup
    • S3 and remote
  • Does truenas have any role to play.
  • Run a full backup.
  • Questions
  • etckeepr
    • Webmin --> Backup Backup Configuration Files --> Backup now --> Include in backup --> Other listed files .. 
      /etc/.git

Update Strategy (Policy)

Your update strategy depends on the type of server you are running

  • Enable automatic software package updates (as appropriate)
    • Webmin --> System --> Software Package Updates --> Scheduled Upgrades
      • Mission Critical Servers
        • Check for updates on schedule?: Yes, every day
        • Email updates report to: enter your email address
        • Action when update needed: Just notify for security updates
      • Not bothered about servers
        • Check for updates on schedule?: Yes, every day
        • Email updates report to: none
        • Action when update needed: Install security updates / Install any updates

Notes

    • Ubuntu/Linux automatically updates the Kernel and sometimes this needs a system reboot. This is not controlled by Virtualmin or Webmin.
    • Automatic/Scheduled Software Package Updates - are they recommended? - Virtualmin - Virtualmin Community
      • Depends on how often you’re in the system. Ideally, you’d pay attention when updates are installed, as updates can break things (though they rarely do).I use automatic updates on systems that I won’t be logging into often, but I usually use the system-provided automatic updates tool rather than the one in Webmin (you can just install unattended-upgrades on Debian/Ubuntu, for example: UnattendedUpgrades - Debian Wiki 1 or dnf-automatic on RHEL and derivatives: Chapter 7. Automating software updates in RHEL 9 Red Hat Enterprise Linux 9 | Red Hat Customer Portal).
      • For systems I am logging into regularly, or that are critical, I run the updates manually, and I make sure I pay attention to security-related updates for packages I’m using, so that I intentionally visit all my non-automatically updating systems to update when major issues arise.
      • In short: If you will make it a practice to become aware of security updates (subscribe to the necessary mailing list(s) for your OS, for instance), then the safest option is to upgrade manually, watch the log of packages to make sure there are no errors, and test immediately after upgrades to be sure everything is happy. But, because it is very dangerous to run unpatched systems, automatic updates are the better choice if you won’t be proactive about updates and becoming aware of security issues in the wild when they come up.
      • Daily updates is reasonable.

Done!!!

  • Install your Client websites.
  • Do a manual backup.

 

 

 

 

Published in Web Server
Read more...
Sunday, 05 November 2023 18:50

My SQL Notes

 

Published in MySQL
Read more...
Sunday, 05 November 2023 17:38

Windows Subsystem for Linux (WSL)

 

Published in Windows 10
Read more...
Sunday, 05 November 2023 16:00

Cannot ping a Windows PC

  • By default the Windows firewall
    • blocks ping requests.
    • blocks traffic from IPs on a different subnet.
  • How to Allow Ping through the Firewall in Windows 10
    • No one can send ping requests to your system to know whether it is alive or not when the Firewall is enabled on your Windows 10 computer system. You can enable ping by disabling the Firewall but this can prove to be very disastrous, as your PC will be exposed to external threats and malware.
    • Step by step instructions with pictures.
    • 'File and Printer Sharing (Echo Request - ICMPv4-In)' / Profile = Private :: I think this will only do local networks (i.e. private)
  • Configure the Windows firewall to allow pings - To configure your firewall to allow pings, follow the appropriate instructions below.
    1. Search for Windows Firewall, and click to open it.
    2. Click Advanced Settings on the left.
    3. From the left pane of the resulting window, click Inbound Rules.
    4. In the right pane, find the rules titled File and Printer Sharing (Echo Request - ICMPv4-In).
    5. Right-click each rule and choose Enable Rule.
Published in Windows 10
Read more...
Sunday, 05 November 2023 09:29

My cPanel Notes

These is a collection my cPanel notes.

General

  • Using FlashFXP to upload files causes my IP to get black listed
    • when i upload a whole website or an operation in flashfxp with a lot of files my IP gets blacklisted
    • This is casued by a misconfiguration of the server's firewall where it is not closing the connection after each file upload. You would see errors in the firewall log simliar to below: 
      Jan 29 10:49:11 host33 lfd[495799]: (CT) IP 146.199.161.166 (GB/United Kingdom/166.161.199.146.dyn.someisp.net) found to have 119 connections - *Blocked in csf* for 1800 secs [CT_LIMIT]
    • Support quote "As per the logs there were 119 connections found in 1800 secs from this IP address which exceeded the connections limit under the csf firewall and the IP address 146.199.161.166 got blocked on the server firewall."
    • Cause
      • Support Quote: "The firewall configuration is set correctly on the server. Sometimes TIME_WAIT connections are triggered under the csf firewall and due to which it detects the more number of connections from the IP address due to which IP address gets blocked at the server end. "
    • Solution
      • Support Quote "We have done port changes at host level can you please try and let us know if you are still facing any issue. Also please let us know if you are using Passive or Active connection of your ftp client."
      • I can now FTP up all of my files and I did not get blocked by the firewall.
  • Do I need CGI?
    • Q:
      • Do i need CGI, does anyone use this normally?
      • I am performing housekeeping and on all of my accounts and there is a folder cgi-bin what is its purpose etc.
    • A:
      • Hoster Response: The cgi-bin directory is used to contain CGI scripts which are rarely used nowadays but can be called, usually using the Perl coding language. However, since the management of content on your site is beyond the scope of our responsibilities I cannot tell how important the cgi-bin folder is for your domains in particular. Depending on whether or not your site uses the directory or CGI scripts deleting the cgi-bin can either break a site or have no effect. Also, cPanel may automatically regenerate the cgi-bin directory if it is deleted. Usually the cgi-bin takes up little or no space on the server so there is little need to remove it.
      • Disable automatic cgi-bin generation | cPanel Forums - I see no reason to have .cgi-bin in any of my sites. Please stop making it a default.
        • You can browse to "WHM Home --> Server Configuration --> Basic cPanel & WHM Setup" and set the following option to "No":
        • Automatically create a cgi-bin script alias. This setting can be individually overridden during account creation.
        • Also, as far as CGI access, you can disable the following options for your packages via "WHM --> Packages --> Edit a Package": CGI Access"
      • CGI Script Alias | Web Hosting Talk - Can i disable CGI Script to prevent virus issue ?
        • You can add the following line inside the global area to disable CGI for all domains. 
          Options -ExecCGI
        • To disable it for all the domains on your server, edit the Apache configuration file 
          pico /usr/local/apache/conf/httpd.conf
        • Search for the line 
          Options -Indexes FollowSymLinks MultiViews
        • in "<Directory "/home">" section and add the following at the end 
          -ExecCGI
        • This should disable it for all the accounts. And yes, you can set n - No for the new accounts, however, the above mentioned changes will disable cgi for the newly created accounts as well.
        • Will it create any issues for working websites if i disable it? Yes, it will. The .pl and .cgi files will not work.
  • Allow SSH connection
    • Add your IP into the hosts.allow file.
    • WHM --> Home --> Security Center »Host Access Control section.
  • To enable Brotli on your cPanel/WHM server:
    • Brotli support in cPanel/EasyApache 4 - PlotHost
      1. Log in to your WHM panel as root.
      2. Navigate to Software->EasyApache 4
      3. Click the Customize button
      4. Now on the Apache Modules tab, search and select mod_brotli.
      5. Click the Next button few times and in the end click Provision button.
      6. You will see the confirmation message The provision process is complete.
  • OS Upgrade
    • cPanel elevate documentation - The cPanel ELevate Project provides a script to upgrade an existing cPanel & WHM CentOS 7 server installation to AlmaLinux 8 or Rocky Linux 8.
  • How to enable terminal and SSH
    • You would be able to get Shell terminal access of server from WHM panel from following steps.
    • Login to WHM --> Home --> Server Configuration --> Terminal
    • If you want to directly get ssh access of server from any SSH client then do let me know your local network's static IP address. So that I would be able to set that in ssh whitelist on your server's /etc/hosts.allow file.
  • Fix Kernel care failing to check for updates etc..
    • Add the following allow IP rule to the firewall (i.e. just add the IP via quick allow) 
      69.175.106.203 # Manually allowed: 69.175.106.203 (US/United States/patches1.kernelcare.com) - Mon Mar 9 17:32:01 2020
  • Change cPanel theme
    • Menu --> cpanel --> customization --> Customize Style --> Basic = set as default
      • This changes it for all accounts as they are set to use default (usually) unless changed.
    • I want to hide the 'Switch to Glass' option for existing customers.
      • You can disable "Change Style" in WHM --> Feature Manager for your feature list(s). That's the only way I see now unfortunately to disable this unfinished theme.

Transfer, Backup and Restore Accounts

  • WHM cPanel account restore
    • Transfer or Restore a cPanel Account | cPanel & WHM Documentation - This interface lets you perform a transfer or restore for a cPanel account via an account archive file.
      • The Transfer or Restore a cPanel Account interface lets you transfer a cPanel account or restore one from an account archive file. An archive file is a cPanel account’s backup file or a cpmove file.
    • How to restore cPanel accounts from WHM - YouTube | PlotHost - How to restore cPanel accounts in WHM.
    • My Instructions
      • Upload the backup file to the `/usr/` folder as there is not much in it. do this as root.
      • I logged in to WHM as root (https://server.yourdomain.co.uk:2087/)
      • Navigate to WHM --> Backup Restoration --> Restore a Full Backup/cpmove File -->
  • HostDime custom backup script
    • This will backup all of the accounts your reseller account owns.
    • The following line had been placed in the root crontab to provide you with client backups: 
      10 0 1 * * /bin/bash /home/.hd/crons/hdbackup_allyourcpanelaccounts.sh | mail -s "Backup cron ran" hosting@yourdomain.co.uk
    • The following is the context of the script "hdbackup_allyourcpanelaccounts.sh" that was customized by Kevin the System Administrator of the time. 
      =========================================================
#!/bin/bash
# HostDime custom backup script
# Author: Kevin B.
# System Administrator

RESELLER=yourreselleraccount
DATE=$(date +"%Y-%m-%d")
LOGFILE=/home/yourreselleraccount/cpanel-backups/logs/$DATE.log
USERLIST=/home/yourreselleraccount/accounts.list
BK_DIR=$(\ls -1 /backup | grep -E "[0-9]{4}-[0-9]{2}-[0-9]{2}" | sort | tail -1)
mkdir -p /home/yourreselleraccount/cpanel-backups/$DATE;
mkdir -p /home/yourreselleraccount/cpanel-backups/logs;

hdbackup () {

## Get a list of users

grep -l "OWNER=yourreselleraccount" /var/cpanel/users/* | cut -d / -f5 > /home/yourreselleraccount/accounts.list

## Skip users that are skipped in backups

echo -e "$(date "+%b %d %H:%M:%S") Verifying users";
while read line;
do
egrep -l "^BACKUP=0|^SUSPENDED=1" /var/cpanel/users/$line | cut -d / -f5 | while read user;
do
sed -i "/$user/d" $USERLIST;
done;
done < $USERLIST

## Tar backups to a folder in the yourreselleraccount account

echo -e "$(date "+%b %d %H:%M:%S") Compressing backups";
cat $USERLIST | while read USER; do
if [[ -d /backup/"$BK_DIR"/accounts/"$USER" ]]; then
echo -e "$(date "+%b %d %H:%M:%S") Copying /backup/"$BK_DIR"/accounts/${USER}";
/usr/local/cpanel/bin/cpuwatch $(grep -c \^processor /proc/cpuinfo) tar -zcf /home/yourreselleraccount/cpanel-backups/"$DATE"/"$USER".tar.gz -C /backup/"$BK_DIR"/accounts/ $USER ;
else
echo -e "$(date "+%b %d %H:%M:%S") Backup does not exist for $USER at /backup/"$BK_DIR"/accounts/${USER}";
fi
done;

## Prune old backups

echo -e "$(date "+%b %d %H:%M:%S") Pruning old backups"
find /home/yourreselleraccount/cpanel-backups/ ! -path '/home/yourreselleraccount/cpanel-backups/' ! -path '/home/yourreselleraccount/cpanel-backups/logs' ! -name "*.log" -mtime +30 -print -delete

## Fix permissions

echo -e "$(date "+%b %d %H:%M:%S") Fixing Permissions";
chown -vR yourreselleraccount: /home/yourreselleraccount/cpanel-backups/

echo -e "$(date "+%b %d %H:%M:%S") Backup Complete";

}
=========================================================
  • The errors bellow are caused by the remote server being blocked by the firewall when transferring cPanel accounts or an incorrect password.
      • [Solved] cPanel copy an account from another server failed | BaseZap - You might have encountered following error while using ” Copy an Account From Another Server With an Account Password 
        Starting “TRANSFER” for “Account” “Username”.
Attempting to copy “Username” from “Source IP”.
Trying to fetch cpmove file via cPanel API!
Fetching current backups from remote server …cPanel Login Failed: 403 Forbidden Access denied
Failed to fetch cpmove file via cPanel API.
Failed: Error while executing “/usr/local/cpanel/scripts/getremotecpmove”. The “/usr/local/cpanel/scripts/getremotecpmove SourceIP Username” command (process 2364424) reported error number 1 when it ended.:
      • Another error 
        TRANSFER: 0 completed, 0 had warnings, and 1 failed.
RESTORE: 0 completed, 0 had warnings, and 1 failed.
TRANSFER: Account “cpanelaccount”: Error while executing “/usr/local/cpanel/scripts/getremotecpmove”. The “/usr/local/cpanel/scripts/getremotecpmove 31.31.31.31 cpanelaccount” command (process 7144) reported error number 255 when it ended.: Cpanel::Exception::HTTP::Network/(XID myb7yt) The system failed to send an <abbr title="Hypertext Transfer Protocol">HTTP</abbr> “GET” request to “https://31.31.31.31:2083/json-api/cpanel?cpanel_jsonapi_module=Fileman&cpanel_jsonapi_func=listfullbackups&cpanel_jsonapi_apiversion=1” because of an error: Could not connect to '31.31.31.31:2083': Connection refused at /usr/local/cpanel/Cpanel/HTTP/Client.pm line 115, <STDIN> line 1. Cpanel::HTTP::Client::request(Cpanel::HTTP::Client=HASH(0x23167b0), "GET", "https://31.31.31.31:2083/json-api/cpanel?cpanel_jsonapi_modul"..., HASH(0x2316930)) called at /usr/local/cpanel/scripts/getremotecpmove line 298 scripts::getremotecpmove::get_current_backups("31.31.31.31", "cpanelaccount", "PUp05bR_Ij%f") called at /usr/local/cpanel/scripts/getremotecpmove line 116 scripts::getremotecpmove::fetch_acct_by_cpanel(__CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__) called at /usr/local/cpanel/scripts/getremotecpmove line 56 scripts::getremotecpmove::script("scripts::getremotecpmove", "31.31.31.31", "cpanelaccount") called at /usr/local/cpanel/scripts/getremotecpmove line 29
RESTORE: Account “cpanelaccount”: Error while executing “/usr/local/cpanel/scripts/getremotecpmove”. The “/usr/local/cpanel/scripts/getremotecpmove 31.31.31.31 cpanelaccount” command (process 7144) reported error number 255 when it ended.: Cpanel::Exception::HTTP::Network/(XID myb7yt) The system failed to send an <abbr title="Hypertext Transfer Protocol">HTTP</abbr> “GET” request to “https://31.31.31.31:2083/json-api/cpanel?cpanel_jsonapi_module=Fileman&cpanel_jsonapi_func=listfullbackups&cpanel_jsonapi_apiversion=1” because of an error: Could not connect to '31.31.31.31:2083': Connection refused at /usr/local/cpanel/Cpanel/HTTP/Client.pm line 115, <STDIN> line 1. Cpanel::HTTP::Client::request(Cpanel::HTTP::Client=HASH(0x23167b0), "GET", "https://31.31.31.31:2083/json-api/cpanel?cpanel_jsonapi_modul"..., HASH(0x2316930)) called at /usr/local/cpanel/scripts/getremotecpmove line 298 scripts::getremotecpmove::get_current_backups("31.31.31.31", "cpanelaccount", "PUp05bR_Ij%f") called at /usr/local/cpanel/scripts/getremotecpmove line 116 scripts::getremotecpmove::fetch_acct_by_cpanel(__CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__) called at /usr/local/cpanel/scripts/getremotecpmove line 56 scripts::getremotecpmove::script("scripts::getremotecpmove", "31.31.31.31", "cpanelaccount") called at /usr/local/cpanel/scripts/getremotecpmove line 29
  • How to Move All cPanel Accounts from One Server to Another | cPanel & WHM Documentation - This tutorial explains how to migrate your cPanel accounts, SSL certificates, and main server IP address from one server to another. Typically, you would do this when you need to replace your server.

Email

  • PDFs are getting stripped from emails when using webmail.
    • You might get an message simliar to this: 
      [Attachment stripped: Original attachment type: "application/pdf", name: "ycc 1581-1.pdf"]
    • Horde -> Attachment stripped | cPanel Forums - You can solve the problem of attachments being stripped from your "sent-box" by simply adjusting your Horde preferences.
      1. Login to your webmail
      2. Click to view your Inbox
      3. Click the "Options" button at the top of the page
      4. Click on "Message Composition"
      5. Look for the following, near the bottom of the list of settings:
        "When saving sent-mail, should we save attachment data?"
        Then set it to "Always Save Attachments" Or any of the other options that suit your personal preference.
      6. Click "Save Options"
  • Email disk Usage ignoring Trash folder
    • SOLVED - Email disk Usage ignoring Trash folder | cPanel Forums
      • Q: How is there 350mb allowed in trash, when i have a limit of 250mb set on the account.
      • A:
        • However, cPanel do not count trash emails in the email account quota as it is excluded by cPanel.
        • It can be enabled from WHM >> Mailserver Configuration >> Include Trash in Quota.
        • But as it is a shared server, it will affect all the accounts on the server so it is not recommended to enable this feature.
        • That is the reason the .Trash folder below is counted in Other Usage instead of email account quota.
  • Emails from ebay and paypal (and other domains) can take ages to turn up, but sometimes they do turn up.
    • Greylisting is enabled
    • The "Bypass Greylisting for Hosts with Valid SPF Records" is not turned on
  • The webmail sub domain (i.e. http://webmail.quantumwarp.com) cannot be accessed, but you can access webmail via https://quantumwarp.com/webmail
    • You might also find other subdomains cannot be accessed and this is most likely becasue your DNS Zone has been corrupted.
    • This sometimes can be caused when migrating betweenb cPanel servers due to differences in version numbers.
    • Solution
      • Backup and customizations you have added to your DNS Zone
      • Reset DNS Zone
      • If this is your primary reseller account you will need to add back in your 'ns1' and 'ns2' entries as these will not be added back in automatically.
      • Re-add your DNS customisations.
  • Have SpamAssassin for non-SPF validated emails
    • Go into SpamAssassin rules and add the following:
      SPF_FAIL = 10     (SPF Hard Failure
  • Some useful notes on spam
    • The spam filter identified spam based on a point value system. Signs of spam such as a blacklisted URL, key words, or little to no verification add points to a messages spam score. Signs of legitimate mail such as proper identification or verification and clear text formatting reduce the spam score. SpamAssassin is currently configured to identify a message spam if it reaches a spam score of 5, though this message received a score of 2.2 with only 1.2 of those points originating from the blacklisting.
    • Blacklisted URLs are not a very effected way to determine if an entire message is considered spam. These rules cause hits regardless of how the URL is provided within the message. For instance, if you were to send an e-mail with a picture taken from the site to show someone it is a scam, SpamAssassin would see the picture is hosted on the blacklisted site, thus giving your legitimate message more points towards the spam score. As a result, we avoid server wide rules for increasing spam scores based on blacklisted URLs alone.
    • As we previously mentioned, we can train SpamAssassin to better identify other parts of the message to increase the spam score. In your case, the Bayes algorithm provided a -1.9 score making it appear more legitimate. This algorithm determines how well this spam message as a whole compares to known spam samples. By training Spam Assassin, the Bayes portion of the filter can identify these types of spam with much higher scores, ensuring the message is properly identified as well as helping prevent future false positives.
  • Horde - large cache of files
    • SOLVED - [CPANEL-12976] Horde generating large number of cache files | cPanel Forums
      • Internal case CPANEL-12976 is open to address the issue where temporary cache files associated with Horde can build up over time in the /home/user/tmp/ (when PHP-FPM for cPanel is enabled) or /home/user/tmp/horde/ directories because they are not automatically removed.
      • The temporary workaround is to manually remove these files, or to setup a cron job to manually remove those specific files after they reach a certain age.
      • Solution
        • In cPanel & WHM version 78, we added the Age, in days, of content to purge users' Horde cache files option to the Mail section of WHM's Tweak Settings interface (WHM >> Home >> Server Configuration >> Tweak Settings). This setting determines the minimum age, in days, of files that the system will automatically delete users' Horde cache files.
        • This setting accepts a minimum value of 1, and defaults to Disabled.
  • Manually train SpamAssassin
    • When our clients are receiving too much spam, we recommend they train SpamAssassin to better identify the type of spam they are receiving.
      • This is done by creating 2 folders using IMAP or webmail, in any email account that falls under the cPanel account that is receiving the excess spam.
      • The 2 folders should be named ".HAM-TRAIN" and ".SPAM-TRAIN", where each of the folders should be populated with at least 200 messages.
      • In the .HAM-TRAIN folder, you should place the legitimate messages received and place the spam messages in the .SPAM-TRAIN folder.
      • Once both folders are populated, let us know so we can perform the training which affects the entire cPanel account, which means this training and folder creation is not necessary to redo on a per email or domain basis
    • The instructions are to move the emails into these folders using the webmail, but can I forward emails to a honeypot email account?
      • In regards to your first question, forwarding messages completely alters the e-mail headers and various sections of the e-mail that may interfere with proper training. Rather than identify incoming spam mail, SpamAssassin may begin to think forwarded mail is spam, thus automatically marking all forwarded mail you receive as spam.
      • Training data is shared across entire cPanel accounts rather than domains or individual e-mail users.
      • We can add the training folders to user@quantumwarp.com and then you simply move the spam/ham messages into their respective folders via webmail or IMAP. Afterwards, we can train using this data and that training data will be used for all domains and all e-mail accounts under that cPanel account.
    • If you would like to copy training data to other domains NOT on the same cPanel account,
      • You will need to copy the two files [bayes_seen] and [bayes_toks] from the SpamAssassin directory within the cPanel account. For example, the account [lancast] has it's training data stored in following two files: 
        /home/yourcpanelaccount/.spamassassin/bayes_seen
/home/yourcpanelaccount/.spamassassin/bayes_toks
      • These files can be copied and moved to other cPanel accounts to share training data.
    • Unfortunately, cPanel does not offer any direct ability to train SpamAssassin, and as such there is little documentation on the topic:
    • For further information on SpamAssassin training, I recommend reviewing the official SpamAssassin training documentation found here:
    • If i use the inbuilt cPanel forwarding feature this should put a copy of the email in another mailbox without altering it so i can then use that spare account via webmail to move spam into the spam folders without affecting my normal work flow?
      • As mentioned previously, we do not recommend setting up a forwarder to send a copy of the messages to another inbox and use the spare inbox to train SpamAssassin.
      • This does alter the message as the message source is now originating from an email account on the server and not the original recipient.
      • The simplest way to fill up your SpamAssassin training folders without affecting your work flow would be to copy the messages from your inbox into the designated SpamAssassin training folders(.SPAM-TRAIN and .HAM-TRAIN), this way you still have the original messages in the folders they were originally in.
    • I am trying to ascertain if a cPanel forwarder is the same as a normal email forward. {see image}. I thought that cpanel just made an exact copy of the email message and effectively copied it and not forwarded it in the traditional sense.
      • A cPanel forwarder is still considered a forwarder where the message headers are altered
    • The training data applies to the entire cPanel account. Each cPanel account under your reseller maintains its own set of training data.
    • You can purge the training data ans start again if you seem to be getting incorrect results
    • The training data on your account can looks like this: 
      ####################
0.000 0 211 0 non-token data: nspam
0.000 0 947 0 non-token data: nham
0.000 0 107557 0 non-token data: ntokens
####################
  • Manually adding SpamAssassin rules
    • The inability to add/remove these rules is simply a limitation in cPanel's UI when viewing the configuration file.
    • These can be manually edited by editing the /home/yourcpanelaccouny/.spamassassin/user_prefs file.
  • Forwarded emails are not getting delivered because they are flagged as SPAM with a 550 error.
  • Increase allowed email size (exim)

SSL

PHP

  • PHP-FPM
  • zlib.output_compression should be disabled
    • This is an option in php.ini settings, and on my server is on by default.
    • Whether to transparently compress pages. If this option is set to "On" in php.ini or the Apache configuration, pages are compressed if the browser sends an "Accept-Encoding: gzip" or "deflate" header.
    • zlib.output_compression Should Be Off on Cloud Server for Performance - zlib.output_compression, Specifically for PHP-MySQL Web Software Like WordPress Should Be Off on Cloud Server for Performance. Here is Why.
    • How to Enable GZIP Compression to Speed Up WordPress Sites - Learn how to enable GZIP compression to speed up your WordPress site on various web servers like Apache, Nginx, and IIS.
  • mime_content_type() function not defined
    • php - mime_content_type() function not defined - Stack OverflowI
      • If you are on shared hosting, chances are that the fileinfo PHP extension is either not enabled or installed.
      • In the case where it's not enabled, navigate to the Software section of CPanel (consult documentation of your control panel if you're not using CPanel) and click Select PHP Version (or something related to that) and enable the extension by checking its box and saving your action.
      • If it's not installed, the extension won't be part of the PHP extensions at cPanel > Software > Select PHP Version > Extensions, edit your php.ini file and uncomment extension=php_fileinfo.dll if you're on Windows. Consult your hosting provider's docs if any of these don't work.
    • Add php73-php-fileinfo to Apache by using  EasyApache

Database

  • This is an example of how our server was tuned using MySQL Tuner
  • We have ran the mysql tuner and as per the suggestions by MySQL tuner, we have changed the MySQL configuration from
  • Before 
    query_cache_size = 48M
query_cache_type = 2
query_cache_limit = 30M
join_buffer_size = 128M
key_buffer_size = 256M
innodb_buffer_pool_size >> unlimited
  • After 
    query_cache_size = 0
query_cache_type = 0
query_cache_limit = 32M
join_buffer_size = 140M
key_buffer_size = 56M
innodb_buffer_pool_size=512M
  • ddddd

 

 

 

 

 

 

Published in cPanel
Read more...
Thursday, 02 November 2023 13:43

pfSense on TrueNAS Scale KVM, What is the best Virtual Custom CPU to choose?

I have been running pfSense (with dedicated quad port card using PCI-E passthrough) for some weeks with no issue as a Virtual Machine on TrueNAS which uses KVM (QEMU). I have been use the 'Custom' CPU option with no model selected which presents the following CPU in pfSense:

QEMU Virtual CPU version 2.5+
4 CPUs: 1 package(s) x 4 core(s)
AES-NI CPU Crypto: No
QAT Crypto: No

NB: QAT = Intel only.

The Problem

MI want to have hardware AES-NI support from the CPU (passed through by the real CPU) but the default QEMU CPU does not have the CPU flags. The other modes don't work either for some reason.

This is what happens when i try the different CPU modes in KVM/QEMU on TrueNAS:

  • Custom Mode (Default/QEMU Virtual) CPU mode
    • Does not support hardware AES-NI (QAT is Intel only) and does not have a lot of the other CPU flags a modern PC has.
    • Exposed to various CPU attacks.
    • pfSense runs fine with this CPU.
    • A very compatible choice, but lacks performance.
  • 'Host Model' CPU mode
    • Allows pfSense to load, but the GUI and routing does not work.
  • 'Host Passthrough' CPU mode
    • Allows pfSense to load, but the GUI and routing does not work.

The issues here are probably caused by one or more of the following:

  • CPU is too new (AMD Ryzen 9 7900 12-Core Processor with 128GB ECC on and x670 board)
  • Being an AMD CPU
  • The OS being FreeBSD (pfSense runs on this OS)
  • FreeBSD driver issues.

The Question

Because my CPU is not compatible, for whatever reason, I will have to select one of the pre-made Custom CPUs (which will adds an emulator layer) but I need one with as many of the features as possible. I am not able to write ans apply my own CPU profile and I would also not want to make changes to TrueNAS manually which is definatley not recommended.

Which one should I choose to get the best out of my CPU?

The Solution

Be aware that as TrueNAS is developed, newer CPU models might become availabe to have greater parity with the QEMU repository.

Conclusions

  • After a brief look of the CPUs supported by TrueNAS, it looked like the all of the newer CPUs, certainly the ones I could identify were server ones.
  • The CPUs on offer are at least 3-4 years older than currently available CPUs.
  • You should use a Custom CPU of the same brand i.e.
    • An Intel Host CPU should use an Intel Guest CPU.
    • An AMD Host CPU should use an AMD Guest CPU.
  • You should choose a Custom CPU that is either the same generation or lower to make sure all the CPU features advertised by the flags can be fulfilled.
  • I do not know what the different CPU modes are
    • -Client
    • -Server
    • -noTSX
    • -IBRS
  • The Best CPU mode selection (in order)
    1. Host Passthrough = This passes the host CPU model features, model, stepping, exactly to the guest.
    2. Host Model = Automatically picks a CPU model that is similar the host CPU, and then adds extra features to approximate the host model as closely as possible.
    3. Custom (Named model) = These allow the guest VMs to have a degree of isolation from the host CPU, allowing greater flexibility in live migrating between hosts with differing hardware.

CPU Selection

Based on my research, my CPU selections are below:

  • Intel
    • Xeon Processor (Cascade Lake, 2019)
    • Xeon Processor (Icelake, 2021/2022)
      • Icelake-Client
      • Icelake-Client-noTSX
      • Icelake-Server
      • Icelake-Server-noTSX
  • AMD
    • EPYC (1st Gen, 2017)
    • EPYC-Rome (2nd Gen, 2018)

Notes

  • If you are not sure if your OS supports a particular CPU, use the QEMU default. It is the most compatible but has security issues. Testing a CPU is always the best way to check compatibility but don't use it on a VM that has live data on it until you are sure.
  • Use the same Brand of CPU as that of the Host Motherboard.
  • You need to use 'CPU Mode = Custom' to use these CPUs.

Research

KVM / QEMU Information

  • The way of KVM: guest's CPU flags | by CocCoc Techblog | Coccoc Engineering Blog | Medium
    • How KVM virtualizes CPU architecture from host machine.
    • The answer is simple: compatibility. By default, KVM sets the cpu mode to custom with generic model— to ensure that a persistent guest sees the same hardware no matter what host the guest is booted on
    • By default KVM uses custom mode and set the CPU model to generic — which misses important flags: aes, pcid and rdrand. If live migration is a concern, use Host model, otherwise, Host passthrough should be used to maximize the features the host’s CPU supports.
    • CPU Modes
      • Host passthrough
        • This passes the host CPU model features, model, stepping, exactly to the guest.
        • Note that KVM may filter out some host CPU model features if they cannot be supported with virtualization.
        • Live migration is unsafe when this mode is used as libvirt / QEMU cannot guarantee a stable CPU is exposed to the guest across hosts.
        • This is the recommended CPU to use, provided live migration is not required.
      • Custom (Named model)
        • QEMU comes with a number of predefined named CPU models, that typically refer to specific generations of hardware released by Intel and AMD.
        • These allow the guest VMs to have a degree of isolation from the host CPU, allowing greater flexibility in live migrating between hosts with differing hardware.
      • Host model
        • This uses the QEMU "Named model" feature, automatically picking a CPU model that is similar the host CPU, and then adding extra features to approximate the host model as closely as possible.
        • This does not guarantee the CPU family, stepping, etc will precisely match the host CPU, as they would with "Host passthrough", but gives much of the benefit of passthrough, while making live migration safe.
  • Qemu/KVM Virtual Machines | Proxmox
    • Qemu (short form for Quick Emulator) is an open source hypervisor that emulates a physical computer.
    • A short but concise overview of QEMU.
  • QEMU User Documentation — QEMU documentation
  • libvirt/src/cpu_map at master · libvirt/libvirt · GitHub - GitHub page with all of the QEMU CPU profiles, if you edit them you can see the CPU flags.
  • Recommendations for KVM CPU model configuration on x86 hosts — QEMU documentation - Seems to be the same as the link below.
  • QEMU / KVM CPU model configuration — QEMU documentation
    • Host passthrough
      • This passes the host CPU model features, model, stepping, exactly to the guest. Note that KVM may filter out some host CPU model features if they cannot be supported with virtualization. Live migration is unsafe when this mode is used as libvirt / QEMU cannot guarantee a stable CPU is exposed to the guest across hosts. This is the recommended CPU to use, provided live migration is not required.
      • It is possible to optionally add or remove individual CPU features, to alter what is presented to the guest by default.
    • Named model (Host Model)
      • QEMU comes with a number of predefined named CPU models, that typically refer to specific generations of hardware released by Intel and AMD. These allow the guest VMs to have a degree of isolation from the host CPU, allowing greater flexibility in live migrating between hosts with differing hardware.
      • It is possible to optionally add or remove individual CPU features, to alter what is presented to the guest by default.
    • Host Model
      • Libvirt supports a third way to configure CPU models known as “Host model”. This uses the QEMU “Named model” feature, automatically picking a CPU model that is similar the host CPU, and then adding extra features to approximate the host model as closely as possible. This does not guarantee the CPU family, stepping, etc will precisely match the host CPU, as they would with “Host passthrough”, but gives much of the benefit of passthrough, while making live migration safe.
    • Default x86 CPU models
      • The default QEMU CPU models are designed such that they can run on all hosts. If an application does not wish to do perform any host compatibility checks before launching guests, the default is guaranteed to work.
      • The default CPU models will, however, leave the guest OS vulnerable to various CPU hardware flaws, so their use is strongly discouraged. Applications should follow the earlier guidance to setup a better CPU configuration, with host passthrough recommended if live migration is not needed.
    • The following CPU models are preferred for use on Intel hosts. See for a list.
      • Intel Xeon Processor (Cascade Lake, 2019), Intel Core Processor (Skylake, 2015).
    • The following CPU models are preferred for use on AMD hosts. See for a list.
      • AMD EPYC Processor (2017).
  • QEMU User Documentation - Xilinx Wiki - Confluence - Seems quite in-depth.
  • CPU Options (-Client/-Server/-noTSX/-IBRS)
  • virtualization - KVM: Which CPU features make VMs run better? - Server Fault 
    kvm -cpu ?model
 x86       Opteron_G3  AMD Opteron 23xx (Gen 3 Class Opteron)
 x86       Opteron_G2  AMD Opteron 22xx (Gen 2 Class Opteron)
 x86       Opteron_G1  AMD Opteron 240 (Gen 1 Class Opteron)
 x86          Nehalem  Intel Core i7 9xx (Nehalem Class Core i7)
 x86           Penryn  Intel Core 2 Duo P9xxx (Penryn Class Core 2)
 x86           Conroe  Intel Celeron_4x0 (Conroe/Merom Class Core 2)
 x86           [n270]  Intel(R) Atom(TM) CPU N270   @ 1.60GHz
 x86         [athlon]  QEMU Virtual CPU version 1.0
 x86       [pentium3]
 x86       [pentium2]
 x86        [pentium]
 x86            [486]
 x86        [coreduo]  Genuine Intel(R) CPU           T2600  @ 2.16GHz
 x86          [kvm32]  Common 32-bit KVM processor
 x86         [qemu32]  QEMU Virtual CPU version 1.0
 x86          [kvm64]  Common KVM processor
 x86       [core2duo]  Intel(R) Core(TM)2 Duo CPU     T7700  @ 2.40GHz
 x86         [phenom]  AMD Phenom(tm) 9550 Quad-Core Processor
 x86         [qemu64]  QEMU Virtual CPU version 1.0
  • How to add a new architecture to QEMU - Part 2 | Florian Göhler - In this article, I will explain how a new CPU can be added to QEMU.
  • Qemu/KVM Virtual Machines - Proxmox VE - A article overviewing QEMU in Proxmox.
  • QEMU/Options - Gentoo Wiki - This article describes some of the options useful for configuring QEMU virtual machines. For the most up to date options for the current QEMU install run man qemu at a terminal.

CPU Information

  • Epyc - Wikipedia
    • Epyc is a brand of multi-core x86-64 microprocessors designed and sold by AMD, based on the company's Zen microarchitecture. Introduced in June 2017, they are specifically targeted for the server and embedded system markets.
    • Epyc processors share the same microarchitecture as their regular desktop-grade counterparts, but have enterprise-grade features such as higher core counts, more PCI Express lanes, support for larger amounts of RAM, and larger cache memory
  • Xeon - Wikipedia
  • List of Intel Xeon processors - Wikipedia

AES-NI / QAT

List of KVM/QEMU CPUs in TrueNAS-SCALE-22.12.3.3

pentium
pentium2
pentium3
pentiumpro
coreduo
n270
core2duo
qemu32
kvm32
cpu64-rhel5
cpu64-rhel6
qemu64
kvm64
Conroe
Penryn
Nehalem
Nehalem-IBRS
Westmere
Westmere-IBRS
SandyBridge
SandyBridge-IBRS
IvyBridge
IvyBridge-IBRS
Haswell-noTSX
Haswell-noTSX-IBRS
Haswell
Haswell-IBRS
Broadwell-noTSX
Broadwell-noTSX-IBRS
Broadwell
Broadwell-IBRS
Skylake-Client
Skylake-Client-IBRS
Skylake-Client-noTSX-IBRS
Skylake-Server
Skylake-Server-IBRS
Skylake-Server-noTSX-IBRS
Cascadelake-Server
Cascadelake-Server-noTSX
Icelake-Client
Icelake-Client-noTSX
Icelake-Server
Icelake-Server-noTSX
Cooperlake
Snowridge
athlon
phenom
Opteron_G1
Opteron_G2
Opteron_G3
Opteron_G4
Opteron_G5
EPYC
EPYC-IBPB
EPYC-Rome
Dhyana
POWER6
POWER7
POWER8
POWER9
POWERPC_e5500
POWERPC_e6500

 

Published in Other Devices
Read more...
Monday, 30 October 2023 09:55

Linux will not boot in a Virtual Machine - Stuck at UEFI Interactive Shell

So you have finished installing your Linux OS on your Virtual Machine and then you get the following error when the system reboots. I have also had a simliar issue when trying to boot of a CD/DVD. You will note this issue only occurs when using UEFI/EFI boot technology and not MBR. You should always use UEFI/EFI where possible.

And just for people search from the internet this is the message in text

Press ESC in 1 seconds to skip startup.nsh or any other key to continue.

This problem seems to only affect some Linux distributions, so far I have experience it with:

  • CentOS
  • Debian

This issue can happen on any Hypervisor but the ones I have had experience with are:

  • KVM
  • VirtualBox

 

The Cause

This is caused by the VM looking in the wrong place for the boot file, or the boot file is in the wrong place, not sure which is more correct but the outcome is the same.

 

Solutions

There are two methods of fixing this which we will go through below. Both allow the system to find the required boot file.

 

Method 1 - Manually add an entry to the UEFI boot order (preferred)

  • Different distros might have different locations for the boot file but the fix should be the same or similar.
  • This should stay after kernel updates.
  • This fix can be done without booting into an OS.

 

  1. When at the interactive shell, type exit

  2. You are now on the Bhyve boot screen
  3. Select 'Boot Maintenance Manager'
  4. Select 'Boot Options'
  5. Select 'Add Boot Option'
  6. You are now in the File Explorer
  7. Select 'NO VOLUME LABEL'. The name might be different on your system.
  8. Select '<EFI>'
  9. Select '<debian>'. The name might be different on your system.
  10. Select 'grubx64.efi'. The name might be different on your system.
  11. You have now finished with the File explorer.
  12. Input the description, in this case 'Debian'.


  13. Commit and Changes and Exit
  14. Change Boot Order
  15. Your screen should look like this, just press enter.
  16. Move 'Debian' to the top
  17. Commit Changes and Exit
  18. Done, you can reboot now.

 

Method 2 - Copy the boot file to the correct location

  • Different distros might have different locations for the boot file but the fix should be the same or similar.
  • This might get wiped out upon kernel updates.
  • You need to boot into the OS to correct this issue using this method.

 

  • When at the interactive shell, type exit

  • You are now on the Bhyve boot screen
  • Select 'Boot From File'
  • Select '<EFI>'
  • Select '<debian>'. The name might be different on your system.
  • Select 'grubx64.efi'. The name might be different on your system.
  • Your VM will now boot up to your selected OS.
  • Now we have booted the VM it is time to put the boot file where it needs to be.
  • Login as root
  • Navigate to the EFI directory of your operating system. 
    cd /boot/efi/EFI/debian/
  • Check you are in the right directory by check to see if grubx64.efi exists. This is the boot file that needs to be loaded but it is in the wrong place.
    ls
  • Make a copy of grubx64.efi into the correct location 
    mkdir /boot/efi/EFI/BOOT/
cp grubx64.efi /boot/efi/EFI/BOOT/bootx64.efi
  • You can now reboot or shutdown your PC. 
    reboot
shutdown

 

Notes

Published in Linux
Read more...
Sunday, 15 October 2023 14:04

Capture VHS Video Cassette Tapes

This article will cover capturing video cassettes to your PC using OBS but can be adapted for other sources.

This guide is aimed at capturing VHS tapes but can be adapted to other analogue tapes

TL;DR

  • Setup
    • OBS Studio
    • Windows 10
    • I-O Data GV-USB2 - Analogue Video Capture dongle
    • NVidia GeForce GTX 1050 Ti OC 4GB
    • Panasonic DMR-EZ48VEBK (DMR-EZ48V)
  • OBS Method 3 - Minimal upscaling to a proper 4:3 resolution
    • Capture video analogue signal with OBS using the I-O Data GV-USB2 via the S-Video port with the following settings:
      • I-O Data GV-USB
        • WEAVE: On
        • Source: S-Video
        • Audio: BTSC
        • Colour Space: Rec. 601
      • Capture Frame Rate and Resolution
        • PAL: 720x576i (5:4) @ 25fps
        • NTSC: 720x480i (3:2) @ 29.97fps
      • Image Processing
        • Deinterlace: Yadif x2
        • Improve Scaling: Lanczos
      • Output
        • Video
          • Encoder: Hardware (AMD/QSV/NVENC, H.264)
          • Rate Control: CQP
          • CQ Level: 23
          • Frame Rate and Resolution
            • PAL: 768x576p (4:3) @ 50fps
            • NTSC: 720x540p (4:3) @ 59.94fps
          • Colour Space: Rec. 709
        • Audio
          • Sample Rate: 48 kHz
          • Channels: Stereo
        • Recording Format: Matroska Video (.mkv)

NB: There are other settings which you need to look at.

My Setup

  • If you are going to buy kit, make sure your GPU or CPU can encode 1080p @ 60fps using H.264
  • You can use software encoder only if you have a fast enough PC.

Everyone's will be different, but this is mine.

My Equipment

  • Windows 10 PC
  • Panasonic DMR-EZ48VEBK
    • Pros
      • S-VHS Quasi Playback (SQPB)
      • S-Video output for the VHS
      • Can turn OSD off
      • Component, RGB and HDMI outputs
      • PAL and NTSC playback of tapes
      • Excellent quality
      • Manuals are easy to find
    • Cons
      • Cannot disable automatic rewind
        • You must play the loaded tape before rewinding (if not rewound) to prevent damage to tape. This procedure allows the player to work out max spin/rewind speed.
      • Cannot disable V.Fast search (fast forward and rewind)
      • The player will stop of there is nothing on the tape.
        • or the counter at least stops
        • is it 5 mins blank = stop ?
  • Daewoo DF-8150P Video Cassette Recorder/DVD Recorder
    • This has a S-Video out.
    • You cannot turn the OSD off.
  • Toshiba DVD Video Player / Video Cassette Recorder SD-23V
    • This has a S-Video out but is only for the DVD player.
  • Nedis VHS-C Cassette Converter (VCON110BK)
    • Do NOT super fast rewind with the adapter.
    • Do not fast forward and rewind tapes with the adapter where possible.
    • I had not choice to rewind the tape in the adapter because I did not have the original camcorder, but what I did was to make sure the rewind mode never went super fast so I was stopping and starting all the time. I did not use the rewind mode with the preview on screen which would have gone slower but the reason I did not use this mode was that I was unsure wether it would harm the tape or not.
    • HOW TO USE A VHS-C TO VHS TAPE ADAPTER - YouTube - Here's how to use a VHS-C tape adapter to watch your old VHS-C tapes in your VHS player.
  • I-O Data GV-USB2 - Analogue Video Capture dongle
  • USB Video Capture Adapter Cable - S-Video/Composite to USB with TWAIN support (SVID2USB232) | StarTech.com (I have not used device so this is for reference only)
    • Comes with Movavi Video Editor 11 SE on a separate CD.
    • When standard driver is installed: Both USB 2828x Device (Video) and USB 2828x Audio Device (Audio) will show up under Sound, video and game controllers.
    • When TWAIN driver is installed: USB 2828x Video will show up under Imaging Device, USB 2828x Audio Device (Audio) will show up under Sound, video and game controllers.
  • Scart --> HDMI Upscaler Converter
  • Rullz 4K 60Hz U3 (USB 3.0 Video Capture with Loop, 3.5 Mic in and 3.5 Audio out)
  • Cables
    • Scart
    • HDMI
    • S-Video
    • Dual Phono to Dual Phone (for audio)

Software

  • OBS (Open Broadcaster Software Studio) - The best open source streaming and capture software.
  • VirtualDub2 - VideoHelp
    • VirtualDub2 (former VirtualDub FilterMod) has all features of original VirtualDub, plus built-in encode/decode of H264 and other formats; open and save MOV, MP4, MKV etc.
    • Video capture software.
  • VLC Player - Everyone's favourite media player.
  • MediaInfo - A convenient unified display of the most relevant technical and tag data for video and audio files.
  • AviDemux
    • A free video editor designed for simple cutting, filtering and encoding tasks.
    • An editor that allows cutting footage without recoding the whole video.
  • MKVToolNix + GUI (gMKVExtractGUI or MKVCleaver)

 

 

Capture Video Cassette Tapes using OBS

There are many ways to sample analogue sources but by far the most used is OBS. These are my settings but can be adapted to match your hardware and setup.

OBS can be complicated for the amateur but once you have been shown around the GUI it is a very easy program to use to capture video and audio from various sources and is not just for Streamers.

  • Don't try to setup or use OBS over Remote desktop as it might causes issues with Sound and Video device mapping issues.
  • OBS only outputs in progressive (1080p, 720p). It will accept interlaced sources (480i, 576i).
  • Use MKV and not MP4. for storage. MKV is a much better format/container and if you need to change this after capture you can.

Setup the PC Environment

  • Update you Windows PC making sure you have the latest video card drivers (no just the ones from Windows Update)
  • Install VLC player
  • Install OBS

Connect up and check your Capture Kit

  • Connect up the capture kit
    • Remember S-Video is better than Composite/RCA
  • Edit the video player's settings for the following (if the option exists)
    • Disable the OSD (On-Screen Display)
    • Set to interlaced output
    • Disable any other image manipulation such as 'comb filter'.
    • Set audio output to Bitstream.
    • Make sure the video output is 4:3 and not 16:9 or auto.
    • Go through all other settings and check the are right for video capturing.
    • Panasonic DMR-EZ48VEBK
      • Menu --> To Others --> Setup --> Picture
        • Comb Filter: Off
        • Still Mode
          • Ignore this as this is just allows you to select wether you show a frame or field when you hit the pause button.
        • Seamless Play: ?
      • Menu --> To Others --> Setup --> Sound
        • Dynamic Range Compression = Off
        • Digital Audio Out
          • Y only need to set this when the using digiatal audio output (SPDIF)
          • LPCM = Audio is output as a left and right channel
          • Dolby Digital/BitStream = Audio is output as digital stream that external kit can decode into audio signal. Supports 5.1
        • PCM Down Conversion: 48KHz
          • Only need to set this when the using digital audio output.
      • Menu --> To Others --> Setup --> Display
        • On-Screen Messages: Off
      • Menu --> To Others --> Setup --> comedian ??????
        • TV Aspect: ? = leavge as is
        • Progressive: Off
        • TV System: PAL (or NTSC if required)

Run the OBS Setup Wizard

When you first open OBS you will need to run through the wizard. The wizard will test your hardware for optimum performance.

It is straight forward, just make your selections as follows:

Step 1 - Usage Information

Step 2 - Video Settings

Step 3 - Final Results

Don't worry if it does show what you expect, we will be changing all of these settings as required.

Create a new OBS Profile and Scene

Creating a separate profile and scene are optional if all you are going to do is capture your VHS tapes and then uninstall OBS, however it does not harm.

A profile is a settings group for OBS and a new profile starts with a lot of the settings at default. A profile also allows you to save your settings for individual projects and export/import them as needed.

  • Menu --> Profile --> New
  • Add Profile
    • Name: VHS Capture
    • Show auto-configuration wizard: unticked

This following just makes things clear and clean so if you are using OBS with other things for example that your scene won't conflict with other scenes.

  • Menu --> Scene Collection --> New
  • Name: VHS

Audio Mixer sources have disappeared

After you have created a new scene the Audio Mixer sources have disappeared, this is normal.

Before

You could have either of these depending what kit you have plugged in.

 

After

Recording Configuration

Encoding

Settings --> Output

There are two options for the 'Output Mode' (Simple' | 'Advanced') which defined what options are available for you to set and which will be left for OBS to set.

  • Simple requires very little configuration and gives really good results.
  • Advanced gives more control and the potential to get much better results.

You need to pick an option and then carry on with the instructions. If you are unsure, start of with using 'Simple' and then move to 'Advanced' when you know what the advanced settings do.

Option 1 - Simple
  • Settings --> Output --> Recording

    • Recording Path: H:\[Your OBS Captures Path]
    • Recording Quality: High Quality, Medium File Size
      • This setting affects the compression level.
    • Recording Format: Matroska Video (.mkv)
      • Don't use mp4. Remux later or have OBS remux the mkv automatically when the capture is finished.
    • Video Encoder: Hardware (NVENC, H.264)
      • Always use H.264
      • If you have hardware encoding support (most modern GPU have this), then select this so your CPU does not need to do the work.
      • Some other possible hardware options
        • Hardware (AMD, H.264) = AMD
        • Hardware (QSV, H.264) = Intel = Quick Sync Video
        • Hardware (NVENC, H.264) = Nvidia = Nvdia Encoding
        • Hardware (NVENC, HEVC) = Nvidia = Nvdia Encoding = H.265
    • Audio Encoder: AAC (Default)
    • Audio Track: Track 1 only
    • Custom Muxer Settings: leave empty
    • Enable Replay Buffer: leave off
    • Notes
      • The bitrate is variable when using `Simple`. The rest of the meta information for the capture you can from the outputted file using MediaInfo.
      • Defaults OBS 'Simple' settings for reference: 
        Recording Quality: High Quality, Medium File Size
Recording Format: Matroska Video (.mkv)
Video Encoder: Software (x264)
Audio Encoder: AAC (Default) / (FFmpeg AAC)
Audio Track: 1
Custom Muxer Settings: {blank}
Option 2 - Advanced
  • Settings --> Output --> Recording Settings

    • Type: Standard
    • Recording Path: H:\[Your OBS Captures Path]
    • Recording Format: Matroska Video (.mkv)
      • Don't use mp4. Remux later or have OBS remux the mkv automatically when the capture is finished.
    • Video Encoder: NVIDIA NVENC H.264
      • Always use H.264
      • If you have hardware encoding support (most modern GPU have this), then select this so your CPU does not need to do the work.
    • Audio Encoder: FFmpeg AAC
    • Audio Track: Track 1 only
    • Rescale Output: leave off
    • Custom Muxer Settings: leave empty
    • Automatic File Splitting: leave off
  • Settings --> Output --> Encoder Settings

    • Rate Control: CQP
    • CQ Level: 23
      • The lower the level:
        • the higher the quality of encoding.
        • the larger the file size.
        • the less compression that is applied.
      • I have chosen level 23 but you might get better results with other values.
      • 15 is practically lossless
      • 0 is lossless
    • Keyframe Interval (0=auto): 0
    • Preset: P7: Slowest (Best Quality)
    • Tuning: High Quality
    • Multipass Mode: Two Passes (Full Resolution)
      • This option controls how and if the encoder pre-scans a frame so it can better compress it.
      • Default: Two Passes (Quarter Resolution)
      • If your GPU or CPU maxes out then you might need to reduce this setting as it can be resource intensive.
    • Profile: high
    • Look-ahead: enabled
    • Psycho Visual Tuning: enabled
    • GPU: 0
    • Max B-Frames: 4
  • Settings --> Output --> Audio --> Track 1

    • Audio Bitrate: 192kb/s
      • Some DVDs and DV use 384kb/s but OBS only goes upto 320kp/s

Advanced Video

The colour space and range need to be defined so the image looks right on modern devices. The defaults are the best but you need to know about these.

  • Settings --> Advanced --> Video

    • Color Space: Rec. 701 (default: Rec. 709)
      • Places like YouTube use 'Rec. 709'and this is the colour space of modern computing and devices. All H.264 stuff should use this colour space.
      • This configures the output, not the input.
      • VHS were recorded in Rec .601 but we are not bothered what they were, but what they will be stored as.
    • Colour Range: Limited (default: limited)
      • When using H.264 you should always use the 'Limited colour Range'.
      • PAL and NTSC only ever used the 'Limited colour Range'.

Audio Sampling

The defaults are the best and most widely used.

  • Settings --> Audio --> General

    • Sample Rate: 48 kHz (default: 48kHz)
    • Channels: Stereo (default: Stereo)

Consider your options

You need to makes yourself familiar with these terms before going further

We will define the properties of the canvas and output here. The following block of text will give you useful information on making your value selections for the different methods.

  • Base (Canvas) Resolution
    • This is the working area of OBS (Scene/Canvas)
    • In normal OBS use, this is the same as your monitor's resolution
    • This area allows you to add multiple streams/sources onto the same output stream/recording. You can move them around to suit your needs such as profile camera feeds/overlays.
  • Output (Scaled) Resolution
    • This is the output resolution of your stream or recording.
    • This has to be the same or larger that the Base (Canvas) Resolution otherwise clipping will occur.
  • Downscale Filter
    • This is the filter that will be used to convert between the Base and Output resolutions if they are different.
    • Bicubic (Sharpened scaling, 16 samples) = Default
    • Lanczos (Sharpened scaling, 36 samples) = Recommended by most people on the internet.

Select your Method

Method 1 - (Digital Source) (Video Downscaling) (Rullz HDMI) - Viewing

Here we are sampling an analogue signal which is then passed to a digital upscaler which is then optionally reduced down to a lesser resolution with 4:3 ratio maintaining the original source's aspect ratio. 

  • I have used this method with my Scart to HDMI Upscaler which takes care of interlacing and outputs a steady upscaled digital stream.
  • The upscaller will only capture at 1280x720p@60fps and 1920x1080p@60fps
  • My Scart to HDMI Upscaler (1920x1080@60Hz) = 1920x1080 @ 60fps
  • I am going to add my Rullz capture device into OBS, your device should add in just the same (except maybe the audio).
  • This alters the video stream because it gets upscaled and filtered, and therefore is no longer the same video stream.
  • Configure the Scene
    • Settings --> Video

    • Base (Canvas) Resolution: 1920x1080
      • Set this to the resolution of your source.
    • Output (Scaled) Resolution:
      • 1440x1080 (4:3)
      • 1920x1080 (16:9)
    • Downscale Filter: Lanczos
      • Depending on your choice above, this might not be needed.
    • FPS: 60
      • or the FPS of your source.
  • Add Video Capture Device
    • Make sure the video is playing a cassette, or at least turned on because the device might auto detect the correct signal.
    • Sources --> Add Source (+) --> Add Video Capture Device
      • Create new
        • Name: Rullz HDMI
        • Make source visible: ticked
    • Set the Properties for 'Rullz HDMI'
      • Device: FHD Video-USB3.0
      • Use custom audio device: ticked

        This option will be missing if you do this over Remote Desktop.
      • Audio Device: Microphone (FHD-Audio)
      • Leave everything else the same

Method 2 - (Analogue Source) (Canvas Rescaling) (Upscaling) (I-O Data GV-USB2) - Viewing

The idea behind this method is to take an analogue source and upscale it to a larger resolution, in this case 1440x1080p (4:3).

  • Configure the Scene
    • Settings --> Video

    • Base (Canvas) Resolution: 1440x1080
    • Output (Scaled) Resolution: 1440x1080
    • Downscale Filter: [Resolutions match, no downscaling required]
    • FPS:
      • PAL: 50
      • NTSC: 59.94
  • Add Video Capture Device
    • Sources --> Add Source (+) --> Add Video Capture Device
      • Create new
        • Name: GV-USB2
        • Make source visible: ticked
  • Configure the Video Capture Device
    • GV-USB2 --> Source Properties --> Configure Video
      • General
      • GV-USB2 - Configure as follows:

        • Vid Deinterlace Method: Weave
        • Video Input: S-Video (or Composite if you only have that)
        • Audio Stereo Sys: BTSC
        • Video Decoder --> Video Standard:
          • PAL_I (UK and Ireland)
          • NTSC_M (North America)
    • Specify the capture resolution (remember, these settings are for the input stream only so they will not affect your recording options)
      • GV-USB2 --> Source Properties
      • Scroll down untill you see Resolution/FPS Type
      • Change the settings as outlined below:
        • Resolution/FPS Type: Custom
        • Resolution:
          • 720x576 (PAL)
          • 720x480 (NTSC)
        • FPS:
          • 25 (PAL)
          • 29.97 (NTSC)
        • Video Format: YUY2
          • The only option I have is YUY2 so Any would work just the same for this setup.
          • Setting this prevents unwanted Video Formats interfering later.
        • Colour Space: Rec. 601
          • This is the one used by legacy TV, Videos and things like that.
          • The OBS default is 'Rec. 709' and that is wrong for this input.
        • Colour Range: Limited
          • OBS default is 'Limited' but there is no harm in setting it here as it is easier to understand.
  • Deinterlacing
  • Resize the capture source to fit the entire Canvas.
    • GV-USB2 --> Right Click --> Transform --> Stretch to screen
  • Increase the quality of the stretch
    • GV-USB2 --> Right Click --> Scale Filtering: Lanczos
  • Filters
    • These are not really to do with cleaning up the video, but more handling green screens and preventing audio spikes, but they could be if the right filter was applied.
    • Filters Guide | OBS - OBS Knowledge Base. A guide to the various effects that can be applied using Filters

Method 3 - (Analogue Source) (Canvas Rescaling) (Minimal Upscaling) (I-O Data GV-USB2) - Viewing (Preferred Method)

This capture type should be used by most people as it keeps as close to the original resolution as possible but deinterlacing it and converting it to a native 4:3 aspect ratio resolution which is suitable for all digital devices.

Follow the instructions from Method 2, but instead use the following Base (Canvas) Resolution and Output (Scaled) Resolution:

  • PAL
    • Settings --> Video

    • Base (Canvas) Resolution: 768x576
    • Output (Scaled) Resolution: 768x576
    • Downscale Filter: [Resolutions match, no downscaling required]
    • FPS: 50
  • NTSC
    • Settings --> Video

    • Base (Canvas) Resolution: 720x540
    • Output (Scaled) Resolution: 720x540
    • Downscale Filter: [Resolutions match, no downscaling required]
    • FPS: 59.94

Additional Settings

  • Enable Monitoring
    • Menu --> Docks --> Stats
    • This will allow you to monitor your PC's resources and make sure they do not get maxed out.
  • Disable/Mute Desktop Audio
    • Do this via the dashboard by clicking on the speaker
    • This prevents notifications and alarms that Windows can generate being added to the recording.

Start Capturing

  • Insert your tape
    • When first inserting a video tape you should play it to:
      1. make sure the video players performs AutoTrack. This stops the process getting recorded in the capture.
      2. check the picture looks good and manually run tracking if required.
      3. you can see the picture in OBS
    • Rewind tape.
  • In OBS click `Start Recording`
  • Press play on the video player
  • When the cassette has finished playing, in OBS, click `Stop Recording`
  • Do a short recording (test run) so you:
    • can check everything is working as expected. You can also check to make sure OBS does not give you warnings about the encoding going faulty because the CPU, GPU or HDD is maxed out.
    • use the Stats Dock to monitor the system resources.
  • Test the recording plays and looks as expected in VLC Player.

 

 

Capture Video Cassette Tapes using VirtualDub2

I have not captured wuith VrtualDub2 so the instructions will be cut down. You need to use VirtualDub2 if you want to maintain the interlaced nature of PAL and NTSC.

Archivists will want to keep the format as close to the original as possible and this is not an issue for playback because interlaced modern TV ans PCs will deinterlace video on the fly.

Select your Method

Method 1 - (Analogue Source) (I-O Data GV-USB2) - Viewing (Preferred Method)

  • This stores the video in a modern format with minimal changes.

This is a modern way of storing your VHS cassettes.

  • Input/Capture
    • I-O Data GV-USB
      • WEAVE: On
      • Source: S-Video
      • Audio: BTSC
      • Video Format: YUY2
      • Colour Space: Rec. 601
      • Colour Range: Limited
    • Frame Rate and Resolution
      • PAL: 720x576i @ 25fps
      • NTSC: 720x480i @ 29.97fps
  • Ouput/Recording
    • Video Encoding
      • Format: AVC (Advanced Video Codec) (H.264)
      • Bitrate: CQP 23
      • Frame Rate and Resolution
        • PAL: 768x576p @ 50fps
        • NTSC: 720x540p @ 59.94fps
      • Recording Format: Matroska Video (.mkv)
      • Chroma subsampling: 4:2:0
      • Video Format: YUY2
      • Color Space: Rec. 709
        • The reason for these changes is that your standard definition video source was originally recorded in a 601 colour space (709 is for HD content and sRGB is for screen captures).
      • Colour Range: Limited
    • Audio
      • Format: AAC LC (Advanced Audio Codec Low Complexity)
      • Sampling rate: 48 kHz
      • Channels: 2 Channels (Stereo)
      • Bitrate: 192kb/s (Some DVDs used 384kb/s)
    • Image Processing
      • Deinterlace: Yadif 2x (Top Field First)
      • Imaging Scaling: Lanczos
  • Post Processing
    • none

Method 2 - (Analogue Source) (I-O Data GV-USB2) - Archiving

  • This maintains the video stream except for changes due to compression by the CODEC (unless you use a Lossless CODEC).
  • The resolution and format will stay the same.

This will create a copy close as possible to the original video. There is no change in resolution or audio settings.

  • Input/Capture
    • I-O Data GV-USB
      • WEAVE: On
      • Source: S-Video
      • Audio: BTSC
      • Video Format: YUY2
      • Colour Space: Rec. 601
      • Colour Range: Limited
    • Frame Rate and Resolution
      • PAL: 720x576i @ 25fps
      • NTSC: 720x480i @ 29.97fps
  • Ouput/Recording
    • Video Encoding
      • Format: MPEG Video
      • Bitrate: Variable
        • Target Bitrate: 3500kbps
        • Max Bitrate: 9000kbps
        • These are a guess for VHS cassettes.
      • Frame Rate and Resolution
        • PAL: 720x576i @ 50fps
        • NTSC: 720x540i @ 59.94fps
      • Recording Format: Matroska Video (.mkv)
      • Chroma subsampling: 4:2:0
      • Video Format: YUV
      • Color Space: Rec. 601
      • Colour Range: Limited
    • Audio
      • Format: MPEG Audio
      • Sampling rate: 48 kHz
      • Channels: 2 Channels (Stereo)
      • Bitrate: 192kb/s (Some DVDs used 384kb/s)
    • Image Processing
      • Deinterlace: n/a
      • Imaging Scaling: n/a
  • Post Processing
    • Edit the MKV and and change the DAR (Display Aspect Ratio) to 4:3

 

 

Capture Video Cassette Tapes using a DVD-RW

This is one of the easiest methods if you have a 'Combi VHS DVD-RW Recorder'.

  • This is the easiest method for anyone and should give good results
  • The interlaced format from the VHS tape will be maintained as per PAL and NTSC formats.
  • These devices will use a Lossey CODEC but usually with a fixed bitrate so it knows how much data it can fit on a DVD.

Instructions

  • Set all recording settings to high
  • Initiate a VHS tape copy
  • Done

 

 

Post Capture Processing

You now have your validated capture you need to make it better.

  • Rename the video file OBS has created (unless you have already changes the output file naming syntax)
  • Trim the unwanted stuff at the beginning and the end using AviDemux.

 

 

Notes

Post Capture Processing

Trimming

MKV

Software

OBS Studio

  • Official Sites
  • General
    • Don't capture over remote desktop as it will mess things up
    • x.264 is the default 'High Quality, Medium File Size' in MKV
    • Can record in x.264/SVT-AV1/AOM AV1
    • x.265/HEVC is not and will not be supported because licensing complexities.
    • default x.264 CODEC settings 
      MKV
Codec: H264 - MPEG-4 AVC (part 10) (avc1)
Encoder: Lavf59.16.100
Codec: MPEG AAC Audio (mp4a)
Channels: Stereo
Sample rate: 48000 Hz
    • interlaced output | OBS Forums
      • OBS does not natively provide deinterlacing, and can only record in progressive scan mode. If your capture card does not provide on-the-fly deinterlacing, you may have to record as progressive-interlaced, then use a video editor to either convert or deinterlace it.
      • Unfortunately OBS cannot output anything but progressive. It's primarily meant as a live production tool, with the secondary ability to record that live content. Many have started using it as an all-purpose recorder, but it is not.
  • Settings
    • To change the codec or output format:
      • File --> Settings --> Output
      • Under the 'Recording' section you can change the output filetype and a few other things.
      • To see more options, set Output Mode to Advanced (this is at the very top)
    • Change file naming convention
      • Settings --> Advanced --> Recording --> Filename Formatting
    • Advanced OBS settings: What they are and how to use them | by Andrew Whitehead | Mobcrush Blog - Ready to take the next step in knowing way too much about OBS? includes B-frames
    • Export Settings
      • Menu --> Profile --> Export
    • Input/Output Resolutions
      • The recording resolution will be the Output (Scaled) Resolution and not the sources input. This is because OBS can have many inputs.
      • Base (Canvas) Resolution = This is the work area, the canvas, where you can arrange multiple sources such as overlays, facecams and your main stream. This is usually the size of your monitors resolution or your primary stream of different.
      • Output (Scaled) Resolution = The resolution of the outputted recording or stream will be.
      • How to Change OBS Output Resolution for Streaming & Recording - YouTube | tech How - A short tutorial on how to change your OBS Studio output resolution for streaming and recording.
    • Remove the red border around the Preview/Canvas/Source window.
      • Question / Help - Red border around sources | OBS Forums
        • Q: When you select a source it shows the boundaries or borders with a red border around the source, is there a png for that or is it something else then a png
        • A:
          • The red border is for sizing the source..Just click on one side and drag it to the size you want..
          • To get rid of the border and lock the source so it's not accidentally moved click the padlock icon next to the source in the source list.
        • Q: I disabled the red borderlines so i cant move anything (it was an accident) But how do i enable it again so i can see the red borderlines so i can move my text again ?
        • A:
          • Click the padlock icon next to the source in the source list. ( unlock it) then it may be necessary to click in the preview window to get the red border to reappear.
          • Just worked it out, go to `Menu --> Edit --> Lock Preview` and untick it:
    • Simple Vs Advanced settings
    • Reset OBS to default settings
    • Automatically Stop recording
    • Colour Settings
      • I-O DATA GV-USB2 default settings sample - VideoHelp Forum
        • Using S-video and OBS right now.
        • Whether this is an issue with the codec, OBS or the card but chromo sub-sampling for NTSC should be 4:1:1. and not 4:2:0 which is PAL
        • Also the DAR appears to be 16:9 and VHS does not support that without letter-boxing.
        • Personally I would do another sample with AmarecTV or Vdub just to compare the output.
        • 4:2:0 colorspace in obs capture is not appropriate, should be 4:2:2
  • Best Settings
  • Capture Tutorials
    • Digitizing VHS Tapes Using OBS - Tim Ford Photography & Videography
      • Did you know you can digitize your VHS tapes using OBS for under $20? Well, you can, and this post tells you how to do it!
      • Has a YouTube Video.
      • Uses the StarTech USB Video Capture Adapter Cable (SVID2USB232)
      • A great tutorial but has a few issues:
        • Fixed bitrate rather than a variable bitrate. Tjhis means somes frames will have their quality reduced. I would recommend variable bitrate so the frame is encoded as required with no comprimise.
        • Uses 29.97 for capturing NTSC rather than 59.94
        • Uses Bicubic and not Lanzcos
        • To get rid of the black bars (overscan) he stretches the image. This will distort the DAR (Display Aspect Ratio). Just leave them in. All tvs (and some modern panels tvs) have overscan of varing amounts.
      • The article explains some other technical stuff including overscan (blackbars), colour profiles and more.
      • Resolutions
        • In the Settings menu, click on the “Video” tab.
          • For NTSC, change the “Base (Canvas) Resolution” to 720×480 (3:2). If you are in the United States, use this setting. The NTSC standard was used in most of the Americas (except Argentina, Brazil, Paraguay, and Uruguay), Liberia, Myanmar, South Korea, Taiwan, Philippines, Japan, and some Pacific Islands nations and territories.
          • For PAL, change the “Base (Canvas) Resolution” to 720×576 (5:4). The PAL region is a television publication territory that covers most of Asia, Africa, Europe, South America and Oceania.
        • If you’ll be digitizing your tapes for use on a modern device (like a computer or a phone) use one of these for the “Output (Scaled) Resolution” setting:
          • For NTSC, type in 720×540
          • For PAL, type in 768×576
        • The reason for this is that your old-school VHS tapes use a resolution that will not look correct when played back on a typical computer or phone screen (it will look a bit stretched). By changing the output resolution, you’ll be using a square pixel aspect ratio which will look correct on more modern devices.
      • Colour Space and Range
        • Go to the “Video” section at the top and change the “Color Space” to “601.” The reason for this is that your standard definition video source was originally recorded in a 601 color space (709 is for HD content and sRGB is for screen captures). The “Color Range” should be set to “Limited.” Press OK.
      • For lossless capture using OBS it'll be 4:2:2, which is technically better than 1394 dv transfers (those are 4:1:1).
      • Recommends
        • CBR
        • 3500 Kbp/s.
        • 48kHz / 192bits
    • Lossless 4:2:2 Digitizing of Video Tapes Using OBS - Tim Ford Photography & Videography
      • Did you know you can digitize your video tapes to lossless quality using OBS? Well, you can, and this post tells you how to do it!
      • Some of the information might not be correct.
    • How To Capture, Denoise, and Restore VHS Tapes - YouTube | TheBenCrazy
      • This video will show you how to record/capture/digitize your old home family VHS tapes or any VHS tapes onto your computer in HD. It will also walk you through using software to denoise and restore the the captured video.
      • This is very thorough tutorial using both Elgato and OBS devices to capture the tapes, it then moves on to showing how to trim the capture with Sony Vegas.
      • All settings are shown.
      • Explains CQP
      • Tells you the best Video Players to buy
    • How to convert VHS videotape to 60p digital video (2023) - YouTube | The Oldskool PC
      • This tutorial will teach you how avoid the most common mistake people make when trying to convert VHS/videotape to digital video -- and all it takes is a $50 piece of hardware and free software. Intended for pure beginners, this tutorial walks you through every step to produce perfect conversions every time.
      • This tutorial uses an Analogue to USB adapter which preserves a lot of the analogue attributes which then need to be dealt with, i.e. interlacing.
      • Explains interlacing
      • Why you should use 60fps
    • Standard Recording Output Guide | OBS - While OBS Studio is strong for broadcasting live to the internet, it is also a great tool for being able to record, either at the same time as streaming or solely for offline usage. 
    • Quick Start Guide | OBS - OBS Knowledge Base. A quick introduction to OBS Studio that guides you towards creating your first stream or recording!
    • Using OBS to Capture Videotapes with a USB Capture Device on Windows - YouTube
      • I have a few issues with this tutorial so do not take all of this process as correct.
      • In this tutorial, I cover the equipment, software, and settings needed in order to successfully capture video from your old, analog videotapes using OBS.
      • Uses the Startech SVID2USB232
      • Settings --> Advanced --> Video --> Color Space:
        • 601 is SD colour space
        • 709 is HD colour space
  • Full Tutorials
  • Misc Tutorials
  • Streaming Tutorials
    • OBS Setup Guide | Volume - A guide to setting up OBS for streaming.
    • How to Use OBS Studio for Professional Video Streaming in 2023 - Want to learn how to use OBS Studio for professional broadcasting? Explore powerful features like window capture in this step-by-step tutorial.
    • Getting started with OBS: A beginner's guide - Koytek Wattenberg Media - OBS is an amazing tool for creators, if you want to live stream; record your videos or even do both at the same time. This guide will focus on beginner advice, and a later guide will tackle more advanced advice regarding the use of OBS and the YouTube Live Dashboard.
    • Best TWITCH Stream Settings for Nvidia users! OBS 28.1 BETA PRESETS - YouTube | EposVox
      • The new OBS 28.1 beta is weird... it adds some new NVENC presets, but are they as magical as it seems?! In this video, I test P1 through P7 of the new NVENC H.264 encoder and test it across Lovelace, Ampere, Turing, Pascal, and Maxwell generations to see what the best settings for you would be.
      • The best settings for NVidia cards using H.264
      • Recommended for streaming
        • Preset: P6
        • Multipass Mode: Two Passes (Quarter Resolution)
    • Never worry about Twitch settings AGAIN! AV1 on Twitch | Nvidia CES News & More! - YouTube | EposVox - Twitch streaming will NEVER be the same! Today at CES, Nvidia helped announce a new Twitch feature called "Enhanced Broadcasting" which will allow the streamer to send their own encoding ladder of transcodes to Twitch instead of relying on Twitch's servers. This gives transcoding to streamers who aren't partnered and can help improve quality and reduce latency! Plus the changes that make this happen allow for Twitch to start leveraging HEVC and AV1 encoding and to start supporting 1440p and 4K streaming! Th
  • Desktop Screen Recording
    • How to Record Your Screen with OBS - YouTube | Guiding Tech
      • OBS, or Open Broadcasting Software, is a free and open source tool that is perfect for streaming and recording right on your desktop. If you’re ready to capture your next gaming experience, here’s what you can do!
      • Add Source --> Display Capture
  • Remux With OBS
    • OBS can remux files into MP4 automatically after recording
    • How to convert/remux mkv files to mp4 using OBS - YouTube
      • Not all video editing programs support mkv files, but OBS Studio (Open Broadcaster Software) has a built-in way to convert (or, more accurately, “remux”) mkv files to mp4 files. Here’s how to do it: Open OBS, click File, then Remux Recordings
    • (OBS REMUX) - How to convert MKV Files with OBS - YouTube
      • Converting (or Remuxing) an MKV file in OBS is extremely easy. While this video is directed towards those who are using OBS to record their screen, the concept also applied if you have an MKV file (maybe from the internet) laying around that you need to change to MP4 format.
    • How to convert mkv to mp4 using OBS studio | Remux recordings OBS studio - YouTube
      • In this video I will show you how to convert mkv to mp4 using OBS studio
    • Standard Recording Output Guide | OBS - If you record in a file format that is not mp4 and want to convert it to mp4 for easy use in the video editing software of your choice or to make it easier to upload to social media, OBS has that built in for you. If you click on File then select Remux Recordings and press the … button to select which video(es) you’d like to remux. After that hit the Remux button and OBS will convert your videos for you, once completed it’ll provide a prompt saying so.
  • Resizing
  • Downscale Filter
    • Default downscale filter = Bicubic (Sharpened scaling, 16 samples)
    • Lanczos filter is the best
    • Best OBS Downscale Filter - The Ultimate Resize Comparison - YouTube | Tech Guides
      • Which is the best OBS downscale filter in terms of performance and video quality? In this video, I compare 9 different methods to downscale a live stream in OBS Studio: Bilinear, Bicubic, Lanczos, Rescale Output, and Canvas Resizing. By looking at gaming benchmarks and an objective assessment of image quality (PSNR) I am able to show that downscaling using the Video Tab and the Lanczos filter is the best choice!
      • Very detailed video on downscaling.
      • Use the Lanczos filter for downscaling, it is the best and is recommend by many.
      • There 3 different types of Downscaling are:
        1. Video Rescaling
          • Settings --> Video --> Base (Canvas) Resolution: 1920x1080 - This is your working area (Canvas) which should usually match your monitor's resolution.
          • Settings --> Video --> Output (Scaled) resolution: 852x480 - This is the resolution of the output used for making files and by making this less than your Base (Canvas) Resolution the output will be downscaled.
          • Settings --> Video --> Downscale Filter: Lanczos (Sharpened scaling, 36 samples) - This is the algorythm used to reduce the Base feed to the required Output resolution.
          • This always uses the GPU to downscale.
        2. Encoder Rescaling
          • Set Base and Ouput resolutions to be the same
            • Settings --> Video --> Base (Canvas) Resolution: 1920x1080
            • Settings --> Video --> Output (Scaled) resolution: 1920x1080
          • Settings --> Output (in `Advanced Mode`) --> Streaming --> Rescale Output: 852x480
            • Select a lesser resolution and it will be downscaled.
            • x.264 = CPU
            • AMD HW H.264 (AVC) = GPU
        3. Canvas Rescaling
          • Set Base and Ouput resolutions to be the same. This resolution will be lower than the input source.
            • Settings --> Video --> Base (Canvas) Resolution: 852x480
            • Settings --> Video --> Output (Scaled) resolution: 852x480
          • This causes the video to be clipped on the canvas. So to fix this:
            • Right click on the canvas --> Transform --> Stretch to screen
            • The video now fits the screen.
          • You can select different filters, but we will select our favourite.
            • Right click on the canvas --> Scale Filtering: Lanczos.
          • This always uses the GPU to downscale.
    • Downscale Filter OBS | tips for efficiency
      • Are you looking for a way to downscale your video streams without sacrificing quality? In this blog post, we’ll introduce you to the Downscale Filter for OBS. We’ll show you how to set it up and how to use it to get the best results for your streams. So, keep reading to learn more!
      • The best downscale filter for OBS will vary depending on your specific computer hardware and internet connection. For most users, the “Bicubic” downscale filter will provide the best results.
      • Bicubic: This is the default filter used in OBS. It does a decent job at downscaling but can sometimes create blurry images.
      • Lanczos: This is the best quality filter, but can sometimes take longer to render.
      • What downscale filter should I use for my twitch streams? Generally, if you have high-speed internet connectivity and a good quality webcam, then using the bicubic filter should give you the best results. But if you have slower internet speeds or a lower quality webcam, then the bilinear or Lanczos filters may be better choices.
    • Getting your video settings right in OBS | by Andrew Whitehead | Mobcrush Blog
      • Upgrade your stream settings for visibly better results
      • We all have a basic grasp of terms like 720p and 1080p — the bigger the number, the better the video quality. But when it comes to streaming, sometimes lowering the quality in one area can help boost it in another.
      • This guide will show you how to set up OBS so you can make an informed decision about what video output resolution is best for your content. Other factors like bitrate (read about that here) and frame rate (full guide here) will also impact your stream quality, so be sure to brush up on those concepts too! Let’s get started.
        • Base (Canvas) Resolution
          • This setting determines the resolution of the space you use to layout your overlays in OBS
          • describes how to set
          • Put simply, the Base (Canvas) Resolution is your main video source that your recordings and streams will feed off.
        • Output (Scaled) Resolution
          • The Output (Scaled) Resolution is used when recording (not streaming) in OBS by taking your Base (Canvas) Resolution and flattening it down for the encoder.
          • If you find any of this confusing, and all you care about is live streaming, set the Base and Output resolution to the same size.
        • Downscale Filter
          • Bilinear and Area are the first two options, but at this point, they’re more like legacy settings that you can ignore. They’re very low quality and you lose too much detail using them.
          • The next two are Bicubic and Lanczos, which are both great options, but Bicubic is the better choice if you want to take a little strain off your PC, while Lanczos looks better looking but needs more CPU or GPU cycles.
          • If you stream using NVENC, you should use Lanczos as the filtering will be handled by your GPU’s onboard encoder and will look much better than Bicubic.
        • Why is this useful? Well maybe you have a Base (Canvas) Resolution of 1080p, and then you need to quickly change to a lower stream resolution for whatever reason, but you don’t want to have to resize ALL your overlays and video sources.
        •  This means:
          • use Lanczos where possible, Bicubic is less CPU intensive but does a worse job.
          • Downscale filters seem to be in order of greatness Bilinear --> Lanczos  in the GUI list.
    • Which downscale filter to use? | OBS Forums
      • The processing load difference between bicubic and lanczos is negligible on any hardware that isn't a complete potato with no business even trying to livestream. Ignore the performance delta as it's unspeakably tiny.
      • Normally bicubic is recommended. It's a standard rescale and provides good quality.
      • Lanczos is more of a personal-preference/situational thing; it's normally used for face-cams and other real-life video... it does have a higher sampling count, and OBS' implementation includes a sharpen pass; good for real video, not so much for synthetic video (like gameplay) where you may get some minor over-sharpen artifacting (like halo effects in solid color blocks). But you likely won't even notice unless you're specifically looking for it.
      • Default: Bicubic (especially for full-frame downscales)
      • Face-cam: right-click, Scale Filtering, Lanczos
      • Lanczos made my stream laggy as hell. Went back to bicubic and it works perfect. my upload is 30mbps and my hardware is AMD ryzen 7 3700x and gtx 1660 super. no hardware or ISP limitations so what gives? Lanczos is a turd do not use folks.
  • Misc
  • Troubleshooting
    • Get log files
      • Menu --> Help --> Log Files
    • No Audio
    • Cannot go full screen
      • Best Ways to Fix OBS Not Recording Full Screen - Being an OBS Studio user, you might have several times caught up with OBS not recording full-screen issues. Well, worry not! As we're here with the best solutions for that. Let's have a look at them.
    • Black Screen
      • OBS: Why Is My Screen Black? Try These Fixes - OBS isn’t immune to glitches, and there’s one particular issue that’s plagued Windows users. We’re talking, of course, about the infamous Black Screen. The error typically occurs during live streaming, and there are several possible causes. In this article, we’ll get to the heart of the matter while showing you how to fix it with step-by-step instructions.
    • Encoding Performance Troubleshooting | OBS - OBS Knowledge Base. Learn best practices to solve encoding performance issues

VirtualDub

  • Supports capture of interlaced videos
  • Capturing interlaced video as interlaced - Is it possible - VideoHelp Forum
    • I have been researching the best way to capture VHS to computer and the best minds say to capture the video as interlaced and to not deinterlace the video. Over the years I have been capturing VHS using a Panasonic DV camera, it captures as interlaced but the color space is 4:1:0. I just recently bought an I-O Data USB capture device and it will capture as 4:2:2, but I can't find any software that will capture as interlaced. I have tried VirtualDub and OSB and both seem to only capture deinterlaced (OSB is for sure that way). Vegas 13 Pro capture program does not recognize he I-O data device as a proper device for capture.
    • Likely you didn't configure VirtualDub properly.
      • Under "Video" -> "Capture pin..." you should select 720x480 for NTSC sources and 720x576 for PAL sources. Some devices like old tuner cards need 704 instead of 720 but 720 is the most common.
        Also select the proper color space here. You want YUY2 or UYVY (both are 4:2:2).
      • That should give you an interlaced capture, unless the capture device itself does something funky or the source is simply not interlaced (two fields taken at the same point in time make up a progressive frame).
    • Thanks for everyone's advice. As it turns out VDub was capturing interlaced video all along. I was using GSpot to determine whether a clip was interlaced or not and none of the field order indicators were set in GSpot, so I assumed the clip was progressive.
  • Capturing with VirtualDub [Settings Guide] - digitalFAQ Forum - My guide is a work in (eternal?) progress. Until then, sanlyn's guide is below. HOWEVER , important update to sanlyn's guide below.

VirtualDub2

This is the successor of VirtualDub and fixes a lot of issues. Instructions and other things out there for VirtualDub will be valid for this software.

AmaRecTV

  • Supports capture of interlaced videos (i think)
  • This is good for showing games on PC in a window, you can deinterlace etc..
  • AmaRecTV 3.10 Free Download - VideoHelp - AmaRecTV is a simple and easy Direct Show Video Capture Recording and Preview tool. Requires the AMV Video Codec (trialware $30).
  • If you do try AmarecTV ignore the bit on the VideoHelp's download page that says it "Requires the AMV Video Codec (trialware $30)." The version on that page (v2.31) doesn't need the AMV Codec to run, you just need to press the 'Update Codec List' button on the 'Recording' tab of the 'Config' window to choose from a list of compatible codecs installed on your system.
  • If you're feeling brave (or can understand Japanese) there are a couple of newer versions available if you poke around on the Japanese AmarecTV website. Version 3.10 is the last version (as far as I'm aware) that doesn't require you to buy their AMV Video Codec. Having said all of that, I'm not sure what advantages v3.10 has over the v2.31 on VideoHelp's download page? Both seem to work well. I'd leave v4.?? well alone as it not only does need their Codec but I think I'm right in saying that you need to do a little registry cleaning after uninstalling it before you can install an earlier version again.
  • AmarecTV Tutorial - YouTube | Armaggedun_ - Quick tutorial on how to use AmarecTV. I hear a lot of people can't figure out how to use it, and/or don't know about it. Thought I'd make this video.

VOB/MPEG Header Editors

When you copy a VOB from a DVD make sure you update all headers.

  • DVDPatcher 1.06 Free Download - VideoHelp - (2003) DVD Patcher is a tool to change the video headers in mpg/mpeg2/vob video. Change aspect ratio, framerate, resolution/size and bitrate.
  • Restream 0.9.0 Free Download - VideoHelp - (2003) With Restream you can change many options of a MPEG2 Elementary Stream without re-encoding. Change Aspect Ratio, Framerate, resolution in the mpeg header, correct and remove sequence extension.
  • MPGPatcher 2020.08.14 Free Download - VideoHelp - (2020) MPGPatcher is a command line tool to change video basics (resolution/size, framerate, aspect ratio, bitrate) in mpg-video files. Patches the video headers only, does no reencoding.

Shotcut

HandBrake (might move to DV)

  • HandBrake – Convert Files with GPU/Nvenc Rather than CPU – Ryan and Debi & Toren - In this post, I’ll show how to use this feature in Handbrake and show some comparisons to illustrate the benefits and tradeoffs that result.
  • Tips for Encoding Videos using HandBrake
    • Tips for creating good video encodings or DVD/BluRay rips, specifically when using HandBrake.
    • The tips give concrete instructions for the program HandBrake, which is a freely available, popular, and good tool for encoding videos—if you use it correctly.
    • A very indepth tutorial and does not just apply to HandBrake.
    • Yadif or Bwdif vs. decomb
    • Denoise
      • In short: if you want to preserve film grain, you will need a very high bitrate. If you want a small file, apply denoising to get good image quality at a low bitrate. NLMeans works best.
      • Modern codecs like H.264 are pretty good at keeping quality acceptable even at lower bitrates. However, although these codecs do have a kind of denoising effect at low bitrates, below a certain point this breaks down and the codec makes a mess of it. If you have a noisy video source (e.g., low-quality VHS tapes, a DVD of an old TV show, a film with a lot of ‘grain’), and you cannot afford encoding it at the extremely high bitrate that will correctly preserve all the noise, then it is a better idea to filter out as much of the noise as possible before the actual encoding starts. The codec will then have a much easier job at producing a good image at a low bitrate.
      • Recent versions of HandBrake have two types of denoise filters: the old HQDN3D (has nothing to do with Duke Nukem 3D by the way), and the new NLMeans.
  • Deinterlacing
    • Most effective 2x deinterlacer? | Reddit
      • They are two different algorithms for deinterlacing.
      • I am a big fan of yadif. It is a much simpler deinterlacer, and much faster, and in motion, to me, everything looks as it would look on an actual TV. If it looks wrong in yadif, then it'll look wrong viewing it on an actual interlaced TV, IMHO.
      • But, decomb is an attempt to improve on it further, and it can sometimes get a slightly better result in cases where yadif (and real interlaced TVs) struggle like near-horizontal lines or repeated patterns of fine horizontal lines. Also, decomb is a bit "smarter" in the sense that it can switch into different modes depending on context. Visually to me, though, occasionally this means it leaves a little bit of "combing effect" in the picture where it is only slight, which yadif by its nature never does. On the other hand, yadif by its nature can tend to have a bit of a "smoothing" effect which you may or may not like.
      • Having performance/speed tested bwdif as implemented in the Handbrake nightlies, it's fast and/or parallelizes well with many cores, so it beats Decomb+EEDI2 by an order of magnitude or more. Hopefully, it ends up being the qualitatively superior option that some users are looking for, but that remains to be seen, I don't think I'm qualified to do that testing so I'll have to wait for somebody else to do it.
    • Best Deinterlace Settings? | Reddit
      • The safest bet if you don't know the source is Bob deinterlace, 2x frame rate (I prefer "yadif" to "decomb" but YMMV, decomb is much slower though) but you can do better with film source DVDs which will usually be telecined as 3:2 pulldown so you can do a detelecine first, in most cases auto will work, and then completely disable deinterlacing and it should be crisp.
      • thanks for sharing your knowledge. I got great results for deinterlacing an old interlaced sitcom from dvd source, went with yadif + bob + 2x framerate (59.94) and the motion is so smooth, picture looks great (although a bit softer), and no visible combing. I always thought "decomb + default" was fine, but I apparently didn't know what I was missing :) It's fantastic.
      • For me, I have found that decomb with the preset of EEDI2 Bob works great. Takes a long time though. I have interlaced detection at default and everything set to off.
    • A Complete Guide to Deinterlace Video with HandBrake
      • How to use HandBrake to deinterlace DVD or video? What's the difference of Yadif and Decomb? Is there a simpler tool than HandBrake to deinterlace video? All will be answered in this article.
      • Yadif is a popular and fast deinterlacer.
      • Decomb switches between multiple interpolation algorithms for speed and quality.
      • Interlace Detection, when enabled, allows the Deinterlace filter to only process interlaced video frames.
    • HandBrake deinterlacing settings? - digitalFAQ Forum
      • Use Decomb, EEDI2Bob
      • It's better than Yadif for AA (anti-alias), but still worse than QTGMC.
      • Yadif leaves % of jaggies, not pleasant to watch.
    • HandBrake deinterlacing settings | Reddit
      • When you use 'bob' you have to set the framerate in the video tab accordingly.
      • 50fps for PAL, 59.94 for NTSC, with 'constant framerate' selected
      • Should come out nice and smooth like watching it on a CRT TV.
      • Field order will be automatically detected.

Misc

  • Capture TV/DVD/VCR Free Downloads - VideoHelp - Download free Capture TV/DVD/VCR software. Software reviews.
  • Best software for capturing? - VideoHelp Forum
    • I was reading a post a week ago by someone on here that knows what he's doing. He recommended some program that was the best for capturing. unlike all of the garbage you get from big box mart. For the life of me, I can't find it.
    • For SD capture, you need a capture card that can pass uncompressed YUY2 to AmaRecTV.
    • VirtualDub
    • VirtualDub (or the VirtualDub FilterMod aka. VirtualDub2 fork) is very flexible as far as capture is concerned. But that can also make it more difficult to get set up properly some devices. Many people have good luck with AmaRecTV after giving up on VirtualDub.
    • Some hints for VirtualDub:
      • Do not play the audio while capturing (turn off Audio -> Enable Audio Playback). This cause A/V sync errors with most devices.
      • Do not compress the audio while capturing (audio codecs are usually single threaded and too slow).
      • Do not capture video uncompressed. Disk drives are too slow for this.
      • Do not use lossy high compression video codecs while capturing (MPEG2, Mpeg4 part2, h.264, h.265).
      • Use fast lossless compression codecs like huffyuv, ut video codec, etc.
      • If you still have audio sync problems play around with the sync settings at Capture -> Timing -> Resync Mode. Especially try enabling Do Not Resych Between Audio And Video Streams (which causes more problems than i solves for many devices).
      • And of course, there's all the usual things to try: https://forum.videohelp.com/threads/104098-Why-does-your-system-drop-frames
    • If you do try AmarecTV ignore the bit on the VideoHelp's download page that says it "Requires the AMV Video Codec (trialware $30)." The version on that page (v2.31) doesn't need the AMV Codec to run, you just need to press the 'Update Codec List' button on the 'Recording' tab of the 'Config' window to choose from a list of compatible codecs installed on your system.
    • If you're feeling brave (or can understand Japanese) there are a couple of newer versions available if you poke around on the Japanese AmarecTV website. Version 3.10 is the last version (as far as I'm aware) that doesn't require you to buy their AMV Video Codec. Having said all of that, I'm not sure what advantages v3.10 has over the v2.31 on VideoHelp's download page? Both seem to work well. I'd leave v4.?? well alone as it not only does need their Codec but I think I'm right in saying that you need to do a little registry cleaning after uninstalling it before you can install an earlier version again.

Capture Hardware Troubleshooting

Rullz

  • No sound when capturing using the Rullz (solution will apply to other hardware)

I-O Data GV-USB2 - Analogue Video Capture dongle

  • Missing capture resolutions
    • The v112 driver has issues and is missing various capture resolutions. You need to use the v111 driver instead.
  • Blackbars on the left and right side of the capture stream
    • This is normal and part of the NTSC and PAL specification.
  • Corruption at the bottom of the capture stream
  • Driver
  • Misc
  • Example Captures (move to top)
  • Blue instead of my capture source picture (based on OBS, but will apply to other software)
    • You are supposed to see a blue screen when the GV-USB2 doesn't have a video input signal. If it wasn't working, you wouldn't be seeing that blue screen.
    • Blue screen usually means no signal. Not exactly the same as no device recognised.
    • When just connected by the S-Video cable this might present as a black screen.
  • Black instead of my capture source picture (based on OBS, but will apply to other software)
    • This can be cause by one or more things listed below
    • Have you got the correct source (S-Video/Composite) selected in the GV-USB2 settings?
      • OBS --> GV-USB2 Properties --> Configure Video --> Custom Properties --> Video Input
    • Have you got the "Video Standard" correctly selected i.e PAL_I or NTSC_M for your region?
      • OBS --> GV-USB2 Properties --> Configure Video --> Video Decoder --> Video Standard
    • Have you set the capture resolution?
      • In OBS you need to specify the resolution because (i think)
        • OBS cannot auto detect it,
        • or when not using NTSC, the default resolution for the device is 720x480 (NTSC)
      • So for your settings should look the image below, but for purposes getting an image on screen we are only concerned about the resolution being manually set.
    • Windows Camera Permissions
      • Streaming / Recording / Equipment forum - GV-USB2 Capture Card Stopped Working in OBS - Speedrun
        • I finally figured out that the capture card stopped working because of an update to Windows 10.
        • As of the Windows 10 April 2018 update, version 1803, you need to change a setting to get this capture card to work. With Win10 (and I assume 11) is that the O/S often blocks access to video capture devices, treating them like cameras. You have to give apps access to cameras in the Privacy Settings. 
          • Start Menu -->Settings --> Privacy --> Camera --> App permissions:You need to toggle "Allow apps to access your camera" to on. If it is already on, turn it off and then back on.
        • After that, the GV-USB2 capture card should show up in OBS, or any streaming program.
    • Check your video player is outputting a signal
      • Check your video player is outputting a signal to a TV so you can rule that out. Start off with the composite signal as this is the most robust.
      • It might not be the usb capture device, it could be the video player itself not outputting a signal. try another video player.
      • The video player might also need to detect the TV at the other end of a cable (S-Video, in particular) or it's firmware does not know what to do or where to output the signal? This might only apply to some of the connections such as S-Video.
      • On your video player, only 1 SCART socket might output the signals.
    • The GV-USB2 is not initiated correctly
      • The usb capture device needs to be receiving a signal when it is hooked up for the first time (I think) so it can correctly established the proper protocol to use.
        • Once you have established your video player is working, connect it by Composite and see if it is now fixed!
        • When testing with the VCR, make sure you have a tape playing. The internally-generated menus & "blue back" of most VCRs is a non-standard signal that many capture devices can't recognize.
      • The video player might not start sending a signal until it sees a TV, the GV-USB2 might not turn on until it gets a valid signal.
        • This is more likely to be an issue when using the S-Video rather than composite, but you never know.
        • The solution is to bring the adapter to life using the composite and then switch to the S-Video. The composiote signal should be fully dumb with no device handshaking.

Panasonic DMR-EZ48VEBK (DMR-EZ48V)

  • Maintenance
  • Troubleshooting
  • Please Wait Error
    • This can be caused by a faulty power supply
    • Can indicate a particular section of the video player not powering up correctly, such as the dvd player.
    • Panasonic DVD Recorder DMR-EZ45VEBS - when I switch it on I get either a "Hello" or alternating " Please Wait" message! | justanswer.com
      • Nothing else happens and I cannot get machine to function with either handset or machine control buttons. It won't even eject or power on/ off. I unplug it form power overnight and same thing happened.
      • Several technical options discussed here.
    • Problem with Panasonic DMR-EZ45V — Digital Spy
      • Q:
        • Over the last week when i finalize a disc it has been making a grinding noise, but would work ok.
        • But today it wouldn't finalize, so i switched the machine off at the mains.
        • Now when i turn it on, the display, just says PLEASE WAIT.
        • It has been doing this nfor more than an hour now.
        • Is there a reset button or maybe something i could do to get it to work!!
      • A:
        • I presume a disc is still in it.
        • It will be stuck in an endless loop struggling to initialise a disc it cannot read, so your priority is to try to get that disc removed so the machine can finish booting properly.
        • So - try again from mains switch on. - Wait 2 minutes. DO NOT press any buttons. If it will not get past the 'please wait'.. press and hold the power switch for 12 seconds.
        • This will hopefully switch the machine off.
        • WHEN it is OFF, press STOP and Channel Up buttons on the unit at the same time and hold for about 5 seconds.
        • Dispose of that disc... but examine the recording surface first... Look for evenness of dye distribution. Look for any obvious surface dirt. Look for any obvious surface damage at the point at the end of the burned area - if there is any. [ Discs are burned from the inside first toward the outside of the diameter. ] ... and note which make and batch it came from.
        • You will very likely find that if you can rescue this situation that most of the disks from that batch will behave similarly.
    • Help with "PLEASE WAIT" message on Panasonic DMR E85H | AVS Forum
      • Replace the HDD
      • Try holding the Channel Up and Down buttons on the unit at the same time. If you can get it to reset, set your clock manually and turn DST OFF.
      • I recently had my first problem with my Pana 85 in a couple of years. This came after one of those very brief power outages. I was getting the "Please wait" and U99 messages, and the unit would only stay on for a minute at a time. The advice from the manual didn't work, so I thought I'd try to track down something from this forum. I found this thread, and sure enough, holding down these two buttons did the trick!
    • Panasonic DMR-E85 Locks Up on Please Wait - ecoustics.com
      • I fixed my DVD Recorder, it turned out to be a power supply issue. There are two capacitors that fail in the power supply (the power supply is located under the hard drive holding bracket). I easily observed the failed capacitors because they appeared slightly bloated, with a slight leakage of substance on the top.
    • Panasonic DMR-ES15 - Please Wait !! | Electronics Forums
      • A guide to diagnosing the power supply and diagnising dodgy capacitors.
    • HOW CAN I FIX ERROR CODE U99 IN PANASONIC DMR-EZ45VEBS? | how to mend it .com - Panasonic dvd players
    • This works on my DMR-E100 so may work on the DMR-ES10.
      • Press & Hold the power button until the machine shuts down
      • Then with the machine off press and hold "stop" & "channel up" buttons on the recorder front panel for over 5 seconds. Release both buttons, the machine should turn on and eject the disk.
    • panasonic DMR-EZ45VEBS U61 error code? | how to mend it .com - Panasonic dvd players
      1. It's working checking the capacitors in the power supply and by the DVD drive and under it, if these have popped tops they have cooked and will allow ripple on the supply lines that cause all sorts of problems including U error codes.
      2. Disconnect from mains, remove metal case cover (four silver screws on sides and three black screws on back) Remove metal plate/cover off top of dvd drive (another four silver screws which are tight !). Clean the dvd spindle with isopropyl alcohol and several cotton buds until the buds come away clean. Do the same for the cap (on the inside of the lid) that rests on top of the spindle if it looks dirty. While you are at it, give the laser lens a GENTLE rub with another clean cotton bud. Be carefully on reassembly the edges of the metal cabinet are SHARP ! After re-assembly and subsequent switch-on insert unformatted Panasonic RAM disc abfpd format it.
      3. I put the disc in shiny side up ie upside down and that worked I then put the disc in the correct way and it worked good Luck
      4. After reading this page I tried inserting a blank unformatted dvd and it cleared the recurring error U61 message.
      5. U61 can be caused by a bad laser.
      6. I was frustrated with this U61 error code until I read the comments on this site. I put in a new DVD-R and the recorder reset itself immediately.
      7. I also solved the problem by opening the front flap and pressing the channel up and down buttons at the same time. Machine went into RESET mode, automatically retuned all the stations and it then worked fine.
  • Reviews
  • Manuals
  • Owner ID Pin
    • How do I reset the owner ID on a Panasonic DMR_EX77 please? | justanswer.com - Unfortunately, you cannot unregister the Owner ID on any DMR player. That information is stored on a NAND flash memory chip, and it cannot be reset or erased in any way. You can reset the player to its factory default condition-with the instructions provided in my previous answer-but unfortunately, the owner ID won't be reset.
    • Panasonic DMR-EZ27EB Owner ID & PIN Number reset | AVForums
      • Once the PIN number has been set, you cannot return to the factory preset
      • The Pin number cannot be reset by button pushing.
      • The reason it would cost for such an operation is that it would involve connecting equipment to erase and reprogram an 'eeprom'.
    • Panasonic DMR-HWT130: PIN problem | AVForums
      • This unit has two pin numbers associated with it: The owner identity PIN, and a separate parental control PIN.
      • You probably put a pin number in on original setup for owner identity...but it seems likely that you have never input a parental control pin, so it should still be at the default 0000, albeit you say you have tried that.
      • The parental control pin is only required for titles that have a 'G' next to them in the list of recorded titles. Is it possible that you have encountered such a title for the first time?
      • The requirement for a parental pin can be turned off for all titles (see page 70 of the manual)... but the pin number is required to change this setting (Catch 22).
      • Reset the parental Pin number by:
        1. While the unit is on, press and hold [OK], the yellow button and the blue button at the
          same time for more than 5 seconds. 
          “00 RET” is displayed on the unit’s display.​
        2. Repeatedly press (right) until “03 VL” is displayed on the unit’s display.
        3. Press [OK].
          • “INIT” is displayed on the unit’s display.
          • The PIN number for parental control returns to the factory preset (“0000”).​

Toshiba DVD Video Player / Video Cassette Recorder SD-23VB

  • Tracking issues
    • Auto-Tracking can be turned of in the OSD/Menu.
    • Manually adjusting VCR tracking function. - The Official Dynabook & Toshiba Support Website provides support for various models.
      • Some of Toshiba’s VCRs will attempt to auto track when a tape begins playing.
      • If the tracking point the VCR chooses is still incorrect, or the VCR did not auto track, the tracking can be adjusted manually.
      • On the VCR itself or on the VCR’s remote, there should be two tracking buttons, a plus (+) and a minus (-). Using these buttons, adjust the tracking until the image is to your liking.
    • I have a toshiba vcr/dvd combination macine. There is no tracking button. Is there a way to automatically adjust the tracking? | Fixya
      • Usually VCRs do not offer specifically labelled tracking buttons as such, however they may incorporate tracking into their channel UP/DOWN buttons, both on the front of the main unit and/or remote. Some brands also offer V-LOCK (vertical lock or still image adjustment) (in pause mode during playback) to stabilise the image, reducing vertical jitter, which again can be adjusted as required using the same buttons as used for tracking. In most cases, pressing both CH UP and CH DOWN together while the tape is playing should centre track (revert back to auto tracking) the unit.
    • I am sure if your VCR has channel buttons on it, try pressing either one while a tape is playing. See if it affects the tracking at all. If it does, press both buttons together for 5 seconds or so, then release - auto/centre tracking takes over.
  • Buying Guide
    • auto tracking ?
    • S-Video is for DVD player only.
    • Can turn of OSD.

Daewoo DF-8150P Video Cassette Recorder/DVD Recorder

  • Connection Procedure - This makes sure the video supplies a video signal.
    1. Make sure you connect th video player to a TV via composite (scart might be ok) before you power the unit on. This allows the video to boot correctly.
    2. You can have the S-Video left Connected to your GV-USB2 device. If you are still having issues, make sure the S-Video cable is disconnected from the video player.
    3. Once the video player has been initiated correctly it will work fine. It might only be after a full disconnection from the power that this needs to be done.
  • Troubleshooting General
    • Daewoo DF8150P VHS/DVD Combo - Locked | AVForums
      • A: The display now show the word "LOCK" when we power up the machine or attempt to use it. There is no mention of how to deal with this in the user guide.
      • Q: With some older Daewoo VCRs to unlock required you to push and hold for 5 secs the power button on the front of the machine...with other models you had to do the same but this time using the power button on the remote control.
    • You need to use audio button on the remote to enable HiFi audio. It will stay on mono until you do this. it will reset back to mono when you eject the tape.
    • The options in the menu are limited.
    • When the RGB option is selected, the video player will do the de-interlacing.
  • Get rid of OSD
    • Using the display button on the remote is the only way to get rid of the OSD
    • turn off/disable VCR On Screen Display for capturing - VideoHelp Forum
      • On pretty much every VCR I've ever used turning off the OSD was a matter of hitting the "display" button on the remote of few times to cycle through the OSD options until it all disappears.
      • You did mention the tracking bar which probably means you have some sort of automated tracking turned on. With that enabled anytime there is a jitter in the tape the VCR wants to adjust you'll see the OSD pop up. There should be an option somewhere in the setting to turn off automatic tracking.
    • A Comprehensive Guide to Learn about OSD Timeout - If you are wondering what does OSD Timeout exactly mean? Here's a Comprehensive Guide to Learn about OSD Timeout.
  • Tracking
    • Auto Tracking
      • The automatic tracking function adjusts the picture to remove snow or streaks. It works in the following cases:
        • When a tape is played for the first time.
        • When the tape speed (SP, LP) changes.
        • When streaks or snow appear because of scratches on the tape.
    • Manual Tracking
      • If noise appears on the screen during playback, press the [TRACKING +/-] buttons on the remote control until the noise on the screen is reduced.
        • In case of vertical jitter, adjust these controls very carefully.
        • Tracking is automatically reset to normal when the tape is ejected or the power cord is unplugged for more than 3 seconds.
  • Green tint on picture
    • Green tint on Daewoo DVD recorder with new tv | AVForums
      • Check to see what output the Daewoo is providing.
      • It sounds like it is outputting S Video... Either change it to RGB [preferably] or plug into a socket in the TV that will take S video and switch / configure the TV input as necessary.
      • It turned out the vcr was set to s-video and not rgb , so a quick menu change improved the picture no end
      • A fully wired scart cable can carry an RGB signal - but only if the scart connector at one end is told to output an RGB signal (as opposed to composite) and only if the scart connector at the other end is told to expect an RGB input (as opposed to composite, or s-video).
      • I would guess that your Daewoo PVR was set to only output composite, the TV is set to expect input composite, so the colours were fine (if not particularly clear). The Daewoo DVDR is set to ouput RGB, the TV is not set to expect input RGB (or can't accept RGB on that particular scart socket), so the colours are poor.
  • Buying Guide
    • cannot turn of auto tracking, when triggered causes OSD
    • S-Video works for VHS and DVD
    • cannot turn OSD  off fully, but can cycle it with the remote control
    • output options are great
    • can copy VHS to DVD

PAL/NTSC/SECAM on VHS, DVD and DV Technology

We need to go over some of the technology so you know why you are selecting certain values and will allow to make changes where necessary.

General

  • PAL
    • Phase Alternation by Line
    • Native storage resolution is 720x576 @ 25fps which is not 4:3.
  • NTSC
    • National Television System Committee
    • Native storage resolution is is 720x480 @ 29.97fps which is not 4:3.
  • NTSC vs PAL
    • What's the Difference Between NTSC and PAL? - The differences between NTSC and PAL are significant, and we're still dealing with them. But both are vanishing from new TVs.
    • NTSC vs PAL - Difference and Comparison | Diffen - NTSC and PAL are two types of color encoding systems that affect the visual quality of content viewed on analog televisions and, to a much smaller degree, content viewed on HDTVs.
    • PAL and NTSC are interlaced. This means that it puts up half a picture every cycle (alternate lines), so you only get 25 full frames a second but because method the picture appears as 60 frames a second. (PAL and NTSC have different timings).
    • You need to capture at the de-interlaced FPS and not the standard frame rate this is because a 30fps interlaced delivers a frame rate of 60fps and if you dont then the video will appear choppy.
      • (PAL 50fps, or NTSC 59.94fps)
    • What is the difference between PAL_B, PAL_D, PAL_ G, PAL_ I | vegascreativesoftware
      • There are various versions of PAL, most commonly used method is called PAL B/G, but others include PAL I (used in the UK and in Ireland) and PAL M (weird hybrid standard, which has the same resolution as NTSC has, but uses PAL transmission and color coding technology anyway). All of these standards normally work nicely together, but audio frequencies might vary and therefor you should check that your appliances work in the country you're planning to use them (older PAL B/G TVs can't decode UK's PAL I audio transmissions even that the picture works nicely).  
      • PAL_I (UK and Ireland)
      • NTSC_M (North America)
    • NTSC vs PAL: What are they and which one do I use? - Corel Discovery Center
      • In PAL regions, the standard household outlet uses a 50Hz current, so the default FPS rate was 25. The other primary difference in the two signals is that PAL signal uses 625 signal lines, of which 576 (known as 576i signal) appear as visible lines on the television set, whereas NTSC formatted signal uses 525 lines, of which 480 appear visibly (480i).
  • Misc
    • Both PAL and NTSC effective display resolution is 720x540 when presented on a TV (cathode ray tube - CRT)
      • PAL has overscan = some pixels get cut off to fit this resolution.
      • NTSC has underscan = the image needs to be stretched to fit this resolution.
    • Each horizontal scan line can be sampled at any resolution becasue it is analogue. 720 is seen as the accepted max resolution you can scan the horizontal after this there is no improvement so not many devices will do above 720.
    • There is aways a set number of vertical scan lines.
    • DV videos are 720x576 (SAR) but have a DAR 4:3 set.
    • DVDs are 720x576 (SAR) but have a DAR 4:3 set.
    • super vhs is best + nicam
      • nicam might only be present on commercial tapes and requires another head on the video player.
    • There is a video player head for each field, so 2 heads for a full frame.
    • A square is a square, so when you strect the captured video stream to 4:3 it will look right as this is all the CRT screen does, takes a weoired resolution and stretches it to 4:3 which is the original ratio of the captured image.
  • To change SAR to DAR
    • Stretching or reducing the NTSC/PAL source to a 4:3 resolution on OBS will correct the view ratio to allow saving the image in the correct ratio
    • You can just add a 'Display Aspect Ratio' (DAR) of 4:3 instead which is how DVDs and DV formats do it. This is only possible when stored digitally and in a format that supports DAR.
  • Terms
    • Glossary of Audio & Video Media Terminology | Media College - Definitions and explanations of audio, video and general media terminology.
    • Storage aspect ratio (SAR)
      • The dimensions of the video frame, expressed as a ratio.
    • Display aspect ratio (DAR)
      • The aspect ratio the video should be played back at.
    • Pixel aspect ratio (PAR)
      • The aspect ratio of the video pixels themselves.
      • A Pixel aspect ratio (often abbreviated PAR) is a mathematical ratio that describes how the width of a pixel in a digital image compared to the height of that pixel.
    • Anamorphic
      • I think this is where the DAR does not match the SAR, or the output resolution is not the same as the stored resolution.
      • HandBrake Documentation — Anamorphic Guide
        • Anamorphic in HandBrake means encoding that distorted image stored on the DVD, but telling the video player how to stretch it out when you watch it. This produces that nice, big, widescreen image.
    • Underscan / Overscan
      • How to Fix Overscan and Underscan Between a TV and Computer - Make Tech Easier
        • When you connect your desktop to your TV, you might encounter an overscan problem. Here are some ways to fix the overscan issue on a TV.
        • But there’s a good chance you’ll encounter problems with overscanning, which is when the monitor or TV cuts off the edges of your desktop. The opposite problem is underscan, where the image is too small for the screen.
        • This tendency of TVs is a relic from the olden days of CRT TVs, but thankfully, it can be fixed using a number of methods we have for you here.
      • How to Properly Crop the Overscan in VirtualDub [GUIDE] - digitalFAQ Forum
        • As anybody converting VHS tapes to DVDs/Youtube quickly discovers, the video signal contains a lot of junk on the edges of the screen -- noise not seen when it was played on a television. This is actually an intentional "feature" of traditional video signals, as it allowed broadcasters to hide non-video signal functionality which did present itself as noise. Closed caption data, for example.
        •  That concept has been explained in depth here: https://www.digitalfaq.com/forum/video-capture/315-errors-edges-converted.html
      • Question about capturing VHS and overscan - VideoHelp Forum
        • Q:
          • I was reading this website about overscanning. According to the source, overscanned ares are not visible when you are watching the content on a TV.
          • So should I crop/add black borders (mask) to cover up a few pixels on the edges (and remove head switching noise) or not?
        • A:
          • All about taste, really up to you. If you do, I would fill the boarders with pure black instead of cropping to a weird resolution, assuming you want this on DVD. And replace the head noise at the bottom with pure black is you want.
          • When you burn it to DVD, it will shrink it to 720x576. Then when played on an HDTV, a 4:3 video will be stretched to 788x576 (for 4:3 PAL Material). So keep this in mind, and maybe just keep your VHS captures at 720x576 when you burn them to DVD. Just don't want you taking my advice from your other thread and upscale the video to 788x576 and then put it on DVD which will just shrink it down again, only to be upscaled again.
        • An in-depth discussion about AR (Aspect ration ) and blackbars (overscan)
        • I know of at least two softwares (AviDemux and VirtualDub *with BorderControl plugin) that can add black over top of your overscan without the cropping/adding hassle.
        • Nobody in the industry cares about this small AR difference and it's common practice to just encode the 720x480 frame when making DVDs from analog video tapes.
        • Just about every 4:3 DVD I come across comes with black bar padding to follow ITU.
        • All DVDs that I know include bars for 4:3 content
        • The amount of overscan varies from TV to TV. In the day of CRTs it could be as much as 10 percent at each edge. So of a 704x480 active picture area as much as 70 pixels at the left and right edges, and 50 pixels top and bottom would be cut off. More typical was about 5 percent. This was because CRTs were not good at keeping the picture the right size and centered. They also suffered from many other geometry problems which were less obvious when you couldn't see the edges of the frame. And all these problems varied from TV to TV, with temperature, age, orientation of the TV, etc. Modern fixed panel TVs don't suffer from these kinds of problems but still overscan by 2 or 3 percent at each edge by default.
      • Black Borders / Black Bars
        • Leaving them in fine and normal
        • This is normal
        • The picture in the middle is the coprrect ration
        • The black bars (Overscaling is to allow old TVS (CRt's) that never couls show thefull picture becasue they were curved and this eased that situation)
        • You can remove the black bar by selecting the area, using shift expand it to cover the whole capture area.
        • Some people post-process and cover the sides with a 'real' black bar and then some devices know to remove them from the picture they display.
        • Why Vmix 22 video with black bars at both right and left 
          • The vmix video both input/output screen has black bar at both left & right end. Even the recorded file. Please Can it be cleared?
          • Let me guess - that "SMI Grabber" is an analogue capture device, and what you're seeing is fact that the active line width in traditional PAL/NTSC video is less than the total line width that is captured (eg 720x576 for PAL).
          • Some cameras fill the entire line with picture content, some don't. Consumer cameras often do, and broadcast cameras typically don't.
          • This area at the edges is usually lost in the "overscan" area of a traditional CRT TV, but the way you're using it (as a source in vMix) you are going to see it.
          • The easiest solution is to zoom in very slightly on the X axis (value >1) so that your active picture fills the width of the screen. To summarize, this is an issue caused by a combination of your camera and capture device - not an issue with vMix.
  • Interlacing / Deinterlacing
    • General
      • Modern screens and devices can only show complete frames, they cannot show individual fields. One frame is two fields.
      • All DVDs are interlaced. This is so they match the NTSC or PAL standards.
      • Interlaced sources are only good on CRT Tvs as they will show artifacts on flat panel TVs or monitors, especially in high movement scenes.
      • When you Deinterlace a source, the frame rate needs to double to match the field rate.
      • Understanding Interlacing: The Impact on Image Quality - DigitalGadgetWave.com - Interlacing is a technique commonly used in television and video to display images.
      • Progressive Vs Interlaced Video Encoding: A Complete Guide - Muvi One
        • Progressive vs interlaced video encoding - a complete comparative guide. Know the differences between progressive vs interlaced video encoding.
        • Once the frame is divided into fields, the encoding process involves the sequential transmission of these fields. Rather than transmitting the entire frame at once, interlaced encoding transmits the odd field first, followed by the even field. 
        • This transmission pattern ensures that each field is displayed in rapid succession, creating the illusion of a complete frame to the viewer’s eye.
    • Interlacing Explained
      • Interlaced video - Wikipedia
      • What is deinterlacing? The best method to deinterlace movies | 100fps.com 
        • A great part of this site deals with interlacing/deinterlacing which introduces some of the nastiest interlacing problems like these.
        • Weave/Do Nothing = Show both fields per frame. This basically doesn't do anything to the frame, thus it leaves you with mice teeth but with the full resolution, which is good when deinterlacing is NOT needed. 
        • Bob (Progressive scan)
          • There is also this way: Displaying every field (so you don't lose any information), one after the other (= without interlacing) but with 50 fps.
          • Thus each interlaced frame is split into 2 frames (= the 2 former fields) half the height.
          • As you see, you won't lose any fields, because both are displayed, one after the other.
        • This article discusses many key facts
      • Video Capturing Concepts: Interlacing Examples – The Digital FAQ - Here are some examples of interlaced and non-interlaced video.
      • Welcome Secrets of Home Theater and High Fidelity - See interlacing explained with animated GIFs.
      • A Guide on Interlaced Video - This blog post guides anyone looking to learn about interlaced videos. It covers topics such as what Interlacing is, how it differs from Progressive Video, and the benefits of Interlacing. Furthermore, it also talks about deinterlacing and how to deinterlace a video for streaming.
      • Deinterlacing in OBS Studio with GV-USB2 - YouTube | Fizztastic
        • This video gives the best example, side by side, of the different deinterlacing filters.
        • Capture settings: GV-USB2 (S-Video), 8:7 Aspect Ratio (Point Scaling), 512x448 Output Resolution.
        • The Filters
          • Left column are control inputs [Bizhark (RGB) and Raw footage].
          • Blend2x is visually incorrect because of the missing flashing of the Dash Bar and in other various places.
          • Linear2x produces a flickery image between the two fields.
          • The best filters is in my opinion Retro followed very closely by Yadif2x.
          • The Retro filter produces a very stable image in flickering condition whereas Yadif2x switches fields producing a slight wavy effect in flashing parts. It also leaves artifacts on the next frame of a disappearing sprite.
        • All the other filters in OBS studio (Blend, Discard, Linear and Yadif) all produce a 30 FPS video.
      • Learn interlacing and field order in Premiere Pro - Learn to convert progressive to interlaced video in Premiere Pro.
      • Field Order - Who's on First? by Chris and Trish Meyer - ProVideo Coalition - If you thought most NTSC video ran at 29.97 frames per second, that's only half the story – literally. It actually runs at a speed of 59.94 fields rather than 29.97 frames per second (fps), with pairs of fields “interlaced” to form a complete frame (see the illustration at left). When you shoot footage with [...]Read More... from Field Order – Who’s on First?
      • Interlace, Interleave, and Field Dominance | mir.com
        • This document presents an overview of the features of interlaced video streams which are essential to understand for working with digital video.
        • All DV streams are lower-field-first.
        • If you are ever going to use a DV source for any of your material, you'll want to choose lower-field-first for all of your material.
      • Digital Video Fundamentals - Frames & Framerates (page 2/3): Progressive and Interlaced - AfterDawn: Guides - There are two basic formats for video, progressive and interlaced. Film is a progressive source because each picture fills the entire frame. That means the framerate is the number of individual pictures. Analog video, on the other hand, uses interlaced, or field based, video.
    • Frames / Fields
      • Fields are not complete images.
        • They are only half of an image at a particular point in time.
        • They are not a half resolution full image. Information is missing.
        • Alternating fields will capture odd and then even rows of an image which looks like a comb.
        • 2 fields make a frame.
      • Fields (Top / Bottom)
        • Which field first? When transcoding or just capturing a video with interlacing, you need to know which filed comes first, but they usually are as follows:
          • VHS: Top Field first
          • DVD: Top Field First
          • DV: Bottom Field First
      • Identifying Top/Bottom field in interlaced video | Mistral Solutions - This paper elaborates an approach that can be adopted to determine top/bottom fields in an interlaced video. Knowing the top and bottom field is important if the video is deinterlaced using Field Combination, Weaving + Bob, Discard and other algorithms based on motion detection.
      • All About Video Fields - Lurker's Guide - lurkertech.com
        • This article explains with the help of diagrams fields and frames.
      • Larry Explains Video Interlacing & Deinterlacing - YouTube - This is an excerpt of a recent PowerUP webinar called "Ask Larry Anything." In this short tutorial, Larry Jordan illustrates what video interlacing is, why deinterlacing is necessary and why deinterlacing always degrades video image quality.
      • Fields & Interlacing Part 1/7: Explained - YouTube
        • The first part to an old but still useful course Chris & Trish Meyer created on the subject of fields & interlaced video. This one covers why interlaced video exists, how it is created, and the difference between fields and frames.
        • At the beggining it shows uyou a great example of fileds and frames.
      • Interlaced vs. Progressive Scan - 1080i vs. 1080p - YouTube | Techquickie - What's the difference between 1080i and 1080p? Does it actually matter?
      • How to view fields
        • It is not always easy to see unless there is a lot of movement in the image, but a sign it is there is the combing artifacts
        • Load the video in AviDemux and do a frame by frame scan and it should show you.
        • VLC Player
    • Deinterlacing
  • VHS Resolution
    • What is VHS resolution? — Digital Spy
      • I am trying to find out what the resolution of VHS and S-VHS is. I know that VHS is 250 lines and S-VHS is 400 lines but I don't fully understand this.
      • The VHS recorder is a two head device with the tape wrapped around just over a half of rotating head assembly (the drum);
      • the odd fields of the interlaced 625/525 video are recorded/played back by one head - the even fields by the other;
      • there is a brief period during the recording process when both heads are in contact with the tape.
      • and more technical information......
    • The VHS Format | Media College - Information about the VHS format, including history, specifications, etc.
    • What is the frame rate of VHS? – VideoAnswers
      • Old school cameras that shoot on VHS and Hi8 formats tend to be 29.97fps and motion pictures shot on film tend to be 24fps.
      • Some other video formats have a frame rate of 23.98 to approximate the film look.
    • What is the Resolution after Converting VHS Tapes? | Legacybox
      • When converting a standard VHS videotape to digital video, the quality will resemble that of analog video. This is a breakdown of all the elements that determine video quality.
      • For the short answer, most tapes are digitized at 480p and about 24-29fps. What does that mean? That means each VHS is digitized at about half of the resolution of high definition, and the frame rate is much lower than most TVs’ max refresh rate is.Reddit - Dive into anything
  • Audio
    • LPCM (from my video manual)
      • Select this when connected to a 2 channel digital stereo amplifier. The DVD Recorder+VCR's digital audio signal will be output in the PCM 2ch format when you play a DVD (or VHS tape) recorded with a Dolby Digital (only for DVD) or MPEG soundtrack. If the DVD is recorded with a DTS sound track then no sound will be heard.
    • Bitstream (from my video manual) = USE THIS ONE
      • This is a didigtal stream straight from the tape.
    • PCM
      • Bitstream Vs. PCM For Audio – Which Is Better? - Bitstream and PCM are capable of producing the same audio quality, and the only difference is how your setup decodes the compressed file. Compatibility with devices and supported frequencies are bigger factors to consider than sound and transmission when choosing between PCM and bitstream.
    • NICAM
      • Nicam: Most Up-to-Date Encyclopedia, News & Reviews | Academic Accelerator
        • An in-depth article on NICAM and it's history.
        • Full-size VCRs were already taking full advantage of tape, using an additional helical scan head and depth multiplexing to record a high-quality audio signal diagonally below the video signal. Mono audio tracks (or, on some machines, non-NICAM, non-Hi-Fi stereo tracks) are still recorded on linear tracks, and when played back on a Hi-Fi machine, they will have the same effect as recordings made on that machine. Backward compatibility has been ensured. Non-Hi-Fi VCR. Such devices are often referred to as "HiFi Audio", "Audio FM"/"AFM" (FM stands for "Frequency Modulation"), and sometimes informally as "Nicam" VCRs (Nicam Broadcast Audio Signal (because it was used for recording). It also recorded standard audio tracks, making it compatible with non-HiFi VCR players, and its excellent frequency range and flat frequency response meant it was sometimes used as a replacement for audio cassette tapes
      • Does this require another head in the viodeo player?
      • Is this only available on commercial tapes because you require a special recorderto put NICAM on the tape.
    • Dynamic Range Compression
      • From the Panasonic DMR-EZ48VEBK manual, page 92
        • Dynamic range is the difference between the lowest level of sound that can be heard above the noise of the equipment and the highest level of sound before distortion occurs. Dynamic range compression means reducing the gap between the loudest and softest sounds. This means you can hear dialogue clearly at low volume.
      • Quick Tip: For Best Audio, Turn OFF Dynamic Range Compression and Loudness Controls — Bob Pariseau - Many Audio Video Receivers (AVRs), and some Source devices such as movie disc players, will include Digital Audio processing options for Dynamic Range Compression or Loudness Adjustment.  Should you use them? In a word, No!  Not if your goal is best quality Audio.
      • How does Automatic Dynamic Range Compression work? | Reddit
        • Dynamic compression basically lowers loud and increases soft sounds. (Normal talking no screaming or whispering but for all sounds)
        • Compression for the audio format is basically packing it into a smaller space, lossless(like trueHD) does this in a way that the sound can be unpacked and still stay identical (like .zip files on the computer) while lossy compression (DD, DD+ etc) gets rid of some of the information to pack it even tighter saving storage space/bandwidth.
      • Dynamic Range Compression? | AVForums
        • If your desire is to listen as the Director intended then surely you should have it switched off? I am not sure why they would recomend it being set to 'STD' as it is obviously applying some compression in that mode.
        • Personally I would leave it off but by all means experiment
        • given that i spend my days coding DRC's and other audio algorithms, if you want the biggest difference between speech and explosions turn DRC to off. unless Sony have messed up their coding, any enabling of drc will result in less range between the quietest and loudest moments.
      • Dynamic Range Compression: Techniques, Applications, And Tips | SoundScapeHQ
        • Discover the definition, purpose, and history of dynamic range compression. Explore its advantages, disadvantages, and how to use it effectively in various applications and genres.
        • Introduction to Dynamic Range Compression
        • Definition and Purpose
        • History and Evolution

Legacy Hardware

  • Composite (RCA)
    • Understanding Composite Video Signals - ClearView - Dive into our detailed CCTV guide to understand composite video signals, their components and their crucial role in CCTV operations.
    • Composite video - Wikipedia
      • A gated and filtered signal derived from the color subcarrier, called the burst or colorburst, is added to the horizontal blanking interval of each line (excluding lines in the vertical sync interval) as a synchronizing signal and amplitude reference for the chrominance signals. In NTSC composite video, the burst signal is inverted in phase (180° out of phase) from the reference subcarrier.[7] In PAL, the phase of the color subcarrier alternates on successive lines. In SECAM, no colorburst is used since phase information is irrelevant.
    • Composite Video vs S-video - Difference and Comparison | Diffen
  • Component
    • Component video - Wikipedia
    • what is the difference between rgb and component?? | Official Pyra and Pandora Site
      • Here is the solution (I work as a technical director at a TV station ;):
        • RGB
          • The best and original color system.
          • You have three lines: Red, Green and Blue.
          • (some RGB, such as RGB cables on computers, also need horizontal and vertical sync lines, but the picture itself uses three lines).
        • Component
          • Developed by Sony.
          • Component also uses three lines, but the three lines consist of:
            • Y = Luminance
            • R-Y (or Cr) = Reduced Red
            • B-Y (or Cb) = Reduced Blue
          • Y usually is the green line from RGB, R-Y and B-Y are pure mathematical calculations. Y is the luminance (so, if you only connect Y, you get a nice B/W signal).
          • A component signal can also be YUV, U is a reduced Cb-signal and V is a reduced Cr signal.
          • Component has been developed when RGB only had 5 lines to get the same image quality with only 3 lines.
          • Most people only know the difference between PAL and NTSC as PAL usually being 50 Hz and NTSC being 60 Hz. But there's another difference:
            • As already stated, if you have a composite signal, the color signal is encoded into the luminance (B/W) signal.
            • This encoded signal is called the "color burst".
            • The first guys to develop the television developed this technique using NTSC - but there has been the problem that the colors shifted (on old TVs you had a knob to recalibrate the colors, new TV sets do this automatically for you).
            • The developers in Germany thought of a solution to this problem a came up with a different burst (a mirrored one, to be exactly) so that the TV sets could automatically handle the colors and there's no shifting (so PAL is more advanced than NTSC).
            • The problem now is: The PAL TVs can't decode the NTSC color burst and the NTSC TVs can't decode the PAL color burst - so only the luminance (B/W) signal can be displayed.
          • Using a scart cable
            • When you use a scart cable, you usually connect your DVD Player, PS2 or whatever else using RGB (three lines instead of one).
            • There's no need to decode any colors because they are transmitted seperately.
            • And that's why you have colors on all kinds of TVs (well, they must at least have a RGB scart).
        • Y/C (some VHS-recorders also call it SVHS)
          • All the colors are put together on one line and the luminance gets one line.
          • So we have a total of 2 lines, but we get some loss in the colors (you won't see them, though, they're minimal).
        • Composite (also called CVBS)
          • That's the worst quality. The colors and the luminance are together on one line.
          • The bandwith per line is 5 MHz, the color is encoded (AM) at 4,43 MHz.
          • For those who want to know a little more:
            • The more the contrast changes, the higher is the frequency.
            • (e.g. if you have a striped shirt, you have a high frequency).
            • When you have a contrast change at exact 4,43 MHz, the TV doesn't know whether this is luminance or a color. That's why you have nice flirring colors at shirts with striped shirts ;))
            • And because we only have small bandwith for the colors, they are really blurry at edges.
      • Oh, and none of the four signals here are digital, all pure analog
    • What is better RGB Scart or component? | Reddit
      • Technically speaking what you're referring to are "RGBS" and "YPbPr". SCART is just a connector, and can carry multiple types of video signal. "Component" just means that the video signal is broken into its separate component parts. The most common type of Component video in use by consumers is "YPbPr Component", but professional equipment often uses "RGBS" Component". Computer VGA is a third, similar signal called "RGBHV". "RGBS" means that there is a single separate sync signal. "RGBHV" means that there are separate horizontal and vertical sync signals. There is also "RGsB", or "sync on green", where the sync is integrated into the green signal.
      • RGB and YPbPr are nearly identical in practice. I've seen in claimed that RGB has slightly better color due to the additional processing YPbPr goes through, but the difference is so small that it's nearly imperceptible, and I doubt most people could distinguish them in an A/B test. Your TV has to convert YPbPr to RGB before it can display it, but the higher quality source means very little is lost in the process.
      • RGB works by breaking the red, green, and blue values of the video into separate signals. This is better than something like composite or S-Video because the color data can't interfere with the other colors (short of an improperly shielded cable).
      • YPbPr has the same advantages, but encodes the signal differently. The "Y" in the name is the green connector, which is a Luma signal (the image in black and white, with sync). The "Pb" and "Pr" (blue and red connectors) are the blue and red offsets. Those signals contain the difference between the Luma and their component color, and that color is then calculated from that value. The green value is then derived from the Y value using the Pb and Pr offsets.
      • They're both about the same. RGB has a slightly higher dynamic color range over YPbPr, but it's not likely something most people will notice. However, RGB is limited to 480i.
  • S-Video
    • Should you get an S-Video VCR? Understanding Super VHS / SVHS and S-Video - If you are trying to achieve the best picture quality, get an S-Video VCR.
    • S-Video supplies luminance (luma, monochrome image) and chrominance (chroma, colour applied to the monochrome image) as separate signals which are read directly from the video tape. Unlike the Composite/RCA where the luminance and chrominance signals are sent down the same cable after one of them has been sent through a filter degregating the signal and leading to a phenomenon called Dot Crawl'
    • S-Video - Wikipedia
      • S-Video (also known as separate video, Y/C, and erroneously Super-Video)
      • S-Video did not get widely adopted until JVC's introduction of the S-VHS (Super-VHS) format in 1987
      • In composite video, the signals co-exist on different frequencies. To achieve this, the luminance signal must be low-pass filtered, dulling the image. As S-Video maintains the two as separate signals, such detrimental low-pass filtering for luminance is unnecessary, although the chrominance signal still has limited bandwidth relative to component video.
    • Test Caps - various composite and s-video cables - VideoHelp Forum
      • Here are some screen caps from AVIs of one of our favorite test patterns showing difference between S-video and Composite.
      • Look closely at the boundaries between different color in these caps:
    • Leads Direct - S-Video Wiring - S-Video is a technical specification for the transfer of video information via a 4 pin mini din cable. These leads are sometimes also referred to as 'S-VHS' leads, which is technically incorrect. However, the two names can be used interchangeably to refer to the same type of cable. These leads are commonly used for connecting video sources such as video cameras, PC Video Grabber cards, DVD players etc.
    • S-Video Cable: All That You Need to Know in Cloom Tech - In this article, we’ll talk about S-Video Cable and answer all the questions you may have about the product.
    • S-Video Cables | cmple.com - s-video cables learning center - learn about different configurations and resolutions of Cmple's s-video cable.
    • What Are S-Video Cables and Connectors For? | Home Cinema Guide - An S-Video cable can be helpful in an AV setup. But, what does it do, and when should you use one? This guide explains when to use an S-Video connector.
  • Scart
    • S-Cideo sockets on scart adapters do not provide a proper S-Video signal, it is just the composite/RCA signal patched onto both the luminesance and chroma lines which therefore only gives you the same quality of a composite signal.
    • Has RGB output available. This might be restricted to 480i max resolution but I have not tested this.
    • Does using an S-Video output via a SCART connector improve the output quality of a VCR? - Video Production Stack Exchange
      • The answer is probably no, unless the SCART socket on your VCR is labeled specifically as "S-VIDEO". The fact that SCART connector has S-Video pins does not guarantee that your VCR provides S-Video signal to these pins. A low-end model will simply transmit a composite signal over the luminance S-Video pin and nothing over the chrominance pin.
      • Even my DVD player having both S-Video and SCART sockets doesn't provide S-Video signal over SCART. Only component RGB.
    • The Ultimate Guide to SCART Connectors and Cables
    • Leads Direct | SCART Wiring - Gives pinouts and a description of scart connectors.
  • SVHS (Super VHS / Super Video Home System)
    • S-Video is not SVHS
    • Super VHS is an improved version of the VHS standard for consumer-level video recording.
    • It was about for a short time before DVDs that provided a better quality experience but required specific video players and different type of video tape.
    • S-VHS - Wikipedia
    • The Many Flavors of Super VHS
      • We'll look at the variations of Super VHS format including S-VHS-C, Super VHS-ET and S-VHS quasi-playback.
      • Recording quality of S-VHS-C camcorders competed with Sony's Hi8 format that also had 400 lines of resolution.
      • S-VHS machines were backward compatible with VHS cassettes but S-VHS video recorders were not selling much in the first few years of production.
    • Learn the Difference Between VHS and S-VHS - Free Video Workshop
      • Although the VHS and S-VHS tape formats look similar, their properties aren't. This article explains the difference between VHS and S-VHS.
      • S-VHS ET = best
      • S-VHS ET was developed by JVC to allow S-VHS ET tapes to be played back on non-ET S-VHS VCRs.
  • Identify VHS Cassette tapes
    • VHS Varieties - How to identify VHS Tape Types - EachMoment - VHS tapes stormed in popularity through the 80s and 90s before declining into obscurity with the rapid rise of the DVD. Now streaming sites like Netflix and Amazon Prime are pushing DVDs into the shadows too. But while the VHS tape was popular, there was lots of innovation and not a lot of universality. Different countries and companies were producing their own twist on the technology and so one VCR was not capable of playing every VHS format.
  • Capture hardware, best to worse, for capturing VHS
    • DV --> S-Video --> Direct Video to DVD (via DVD-RW/Video Combi) --> Component (YPbPr) --> Component (RGB) --> Composite (RCA)
      • DV
        • Fully digital so there is no data loss. Do not use analogue methods to capture this.
      • S-Video
        • This is direct supply of each the luminance and chrominance signals from the video tape.
        • This will allow you to process the video on a PC with modern algorithms and methods not present on the video player whoes hardware and programming cannot be changed.
        • This method does not suffer from dot crawl as does composite.
      • DVD-RW/Video Combi
        • This depends on the quality of the hardware as to whether this is better than S-Video.
      • Component (RGB / YPbPr)
        • This signal has been made by converting the luminance and chrominance signals on the video tape and split into components, so it is dependant on the hardware of the device to do a good job. Component does however have a higher bandwidth that composite and S-Video and in other types of capture this might be the preferred method. An edge case would be capturing DVDs but why would you capture these via analogue when they are already a digital format.
      • Composite (RCA)
        • The original and worst technology to use.
  • VHS
    • Chroma and luminesance are stored as separate data streams on the video tape.
      • S-Video provides these streams as separate data giving a better quality capture.
      • Composite carries both these streams over the same cable, but one of them goes through a low pass filter to prevent interference, however this causes a degredation in the signal and phenominen called dot crawl wihich impares the picture quality.
  • DVD
    • DVD-Video - Wikipedia
      • Has the audio and video specs.
    • DVDs have the following attributes
      • 768x576 (4:3)
      • Can store Interlace/Deinterlaced
      • Can specifiy the viewing ratio of the video file which allow hardware to dynamically change the image output as required to show properly.
    • What is DVD? - VideoHelp
      • DVD stands for Digital Versatile/Video Disc, DVDR stands for DVD Recordable and DVDRW for DVD ReWriteable.
      • This article goes into great detail about the technical specs of the DVD.

Pixel Shapes (Square, Thin, Fat)

  • Pixels on CRT TVs were not square, they were usually taller and the technology was aware of this so an image at a set resolution shown on a TV correctly will appear squahes or otherwised strectched when viewed on a monitor with square pixels. This means some extra work needs to be done on the source to get it to show properly on modern displays.
  • When VHS, PAL and NTSC videos are displayed on TVs (CRT) the ratio is 4:3 (DAR) however because CRTs don't use square pixels the the ratio of the video signal (vertical to horizontal) on a VHS is different (SAR).
  • The effective display of both PAL and NTSC is 720x540 (4:3), NTSC is stretched (this might be called underscan) and part of the PAL signal is cropped (the overscan) allowing both systems to have the same viewing output.
  • PAR, SAR, and DAR: Making Sense of Standard Definition (SD) video pixels - BAVC Media
    • By Katherine Frances NagelsIt’s well-known that while motion picture film has seen many different aspect ratios come and go over its history, video has been defined by just two key aspect ratios: 4:3 for analogue and standard definition (SD) video, and 16:9 for high definition (HD) video. Simple, right? Yes—but underlying this are some aspect ratios that are not so straightforward: those of the video pixels themselves.
    • This article successfully explains that PAL and NTSC do not have square pixels and how this can affect rendering of digitally captured analogue videos.
    • We now have two video resolutions: 720×576 and 720×480, and we know that the aspect ratio of the video frame is 4:3. Yet, it’s clear even at a glance that these two dimensions cannot both produce a 4:3 image. A closer look and a quick maths equation reveals that in fact, neither of these frame dimensions are 4:3!
    • And this is where the non-square pixels come in. In effect, SD video is slightly anamorphic: in order to meet the specifications of Rec. 601 and also fill a 4:3 screen, SD pixels are ‘thin’ or ‘fat’.
    • Since it will probably be transferred at 720×486 or 720×576—as is best practice for preservation
    • But 480i pixels are higher than they are wide, with a pixel aspect ratio (PAR) of 10:11. What about 576i pixels? It’s the reverse.
    • Excellent visual comparision between square and thin/fat pixels
  • Pixel aspect ratio - Wikipedia - A Pixel aspect ratio (often abbreviated PAR) is a mathematical ratio that describes how the width of a pixel in a digital image compared to the height of that pixel.
  • About Aspect Ratios
    • We shall talk about three aspect ratios: frame-size aspect ratio (far), the pixel aspect ratio (par) and the the display aspect ratio (dar).
    • All aspect ratios are given as the ratio of width to height of the rectangle.
    • The frame-size aspect ratio is the shape of the data stored.
    • The pixel aspect ratio determines the shape of a pixel.
    • The display aspect ratio determines the shape of the image that will be displayed.
    • This goes into the maths used to create these values.
  • PAL D1/DV Widescreen square pixel settings in After Effects (CS4 vs CS3) | Mike Afford Media
    • Seems the latest version of After Effects from Adobe (CS4) has changed the PAL D1/DV Widescreen square pixel preset. In CS3, compositions using that preset would be set to 1024 x 576 pixels. The new version (CS4) uses 1050 x 576. So which is right? 1024 or 1050?
    • Has visuals to help with this question and shows the different type of pixel shape.
  • Solved: PAL pixel aspect ratio issue - Adobe Community - 13042553
    • I'm working with some old PAL footage, 720x576.  Premier says its PAL pixel aspect ratio is 1.0940; however the correct pixel aspect for this resolution is supposed to be 1.0666.
    • Change the Comp settings to 720x540 Square Pixel for 4:3 and 960x540 Square Pixel for 16:9. Use Layer > Transform > Fit to Comp to fit the PAL Source exactly to the manually set square
    • I found that my footage of 720x576 would scale to be exactly 4:3, using the PAR of 1.06. The adoption of 1.09 is based on 704x576, which is considered [I think] the displayable portion of PAL. So that explains to me why they adopted this value.
    • Change the Comp settings to 720x540 Square Pixel for 4:3 and 960x540 Square Pixel for 16:9. Use Layer > Transform > Fit to Comp to fit the PAL Source exactly to the manually set square pixel frame sizes.
  • Understanding PAL aspect ratio? - digitalFAQ Forum
    • The actual video area usually is not 704x480 either. The exact measure varies. Remember, the source was analog, not digital. It wasn't measured in precise pixels. 720x576 is essentially 704x576 with an added matte. The matte was missing in the 704.
    • Most lossless codecs don't honor DAR on playback, they simply play the frame as-is.
    • The physical aspect ratio of the original 720x576 frame is 5:4, which is not a 4:3 image. VHS and VHS-C are designed for playback as 4:3 images for your old 4:3 CRT TV. As far as rectangles go, a 4:3 image is slighter wider than a 5:4 image. Another way of stating the image ratios is that 4:3 = 1.333 to 1 and 5:4 = 1.25 to 1.
    • 4:3 is the only image ratio that VHS and VHS-C can were designed to play as analog tape source, whether the image has extra borders or no borders and whether the core image fills the entire frame or not.
    • The reason for capturing to the anamorphic format of 720x576 is because that is the format that will be required for DVD or Standard Definition BluRay authoring.
    • You can also crop the 720x576 image to 704x480 (sorry, but a width of 702 simply will not play correctly and your DVD authoring program won't let you use it). ALso, some ornery equipment won't use 704 width exactly, but can use more or less than 704. It depends on the source and the capture gear. If you wanted square-pixel 4:3 for playback from FFv1, you should have encoded to 768x576 or to the more standard 640x480 (note that you would still have side borders and head-switching noise, and neither of those frame sixes can be used for DVD or BluRay).

Ratio, Resolution, PAR, SAR and DAR calculations

This area can be quite tricky to understand but is not needed for most people and is here as a reference for me and other nerds.

  • To get the DAR resolution
    1. You can use media info to get the DAR but it will only show you the ratio. if you put this same file into handbrake it will show you thre actual resolution of the DAR
    2. To get the DAR resolution of a film and not just the ratio, play it in VLC Player and then save a screen shot. This will give you the true DAR.
  • NTSC 4:3 aspect ratio 720x540? - digitalFAQ Forum
    • For uploading to YouTube/sharing, I am using ffmpeg to change the storage aspect ratio and re-encode to H.264 MKV files. This is working fine and I've got no problems.
    • For archiving the original HuffYUV files, I am using ffmpeg to change the display aspect ratio and remux into an MKV. I am changing the DAR only, with the intention being simple playback at the correct aspect ratio with no other changes to the file. SAR is not changed and the file is not re-encoded.
    • This was going fine working with my PAL tapes (I think), but now I've tried NTSC and I'm having difficulties. I've done a lot of Googling over the past few hours but haven't really got a clear answer.
    • Capturing at 720 and cropping to 704, Again it's only about 3% stretch if you keep 720 and don't mind the ugly 16 grey pixels on the sides of the frame. 704 is accurate, 720 is an approximation according to the D1 standard.
  • Is 720x480 DVD source conversion to 720x540 upscaling? - VideoHelp Forum
    • ALL DVDs have a Storage Aspect Ratio (SAR) of:
      • 720x480 for NTSC
      • 720x576 for PAL
    • When the Display Aspect Ratio (DAR) is 4:3, the display is resized to 720x540, for both NTSC and PAL
    • When the DAR is 16:9, the display is:
      • 854x480 for NTSC
      • 1024x576 for PAL.
    • The variances are due to the simple fact that DVD pixels are not stored as square (PAR=Pixel Aspect Ratio) whereas they are displayed square.
    • H.264 has nothing to do with the original DVD. You must be looking at a conversion
    • All NTSC DVDs are 720x480 (well 704x480 is possible for 4:3, but pretty rare).
    • If you keep the 720 width the same and stretch the 480 height until it's 4:3, in square pixels terms you end up with 720x540.
  • DVD, 720*480 or 720*540 | AVS Forum
    • 720x540 is the display size for both of these formats. Both formats have non square pixels.
    • NTSC pixels are stored a little short and fat, PAL pixels are stored a little tall and skinny.
  • Is PAL 720x576 or 768x576 - VideoHelp Forum
    • jagao
      • Analogue PAL is 576 discreet scan lines but on the horizontal axis is a continuous waveform. It can be sample with as few or as many pixels as you want. It is customary to sample with 720. That is generally considered enough to capture all the detail of the highest quality analogue PAL sources, without being excessive.
      • PAL DVDs, for example, use a 720x576 frame. 720x576 is a 5:4 aspect ratio so the image is adjusted at playback to give a 4:3 picture.
      • Whether you want to resize to square pixels depends on what you are making. DVDs don't support 768x576 so you should leave the video 720x576. If you want to upload to Youtube or some other video sharing network you might want to use square pixels
    • Pandy
      • This depends from sampling clock - for 13.5MHz sampling clock there is 720 pixels max, for 14.75MHz there is 768 pixels max.
        BTW 768 is for square pixel (pixel aspect for 4:3 screen is 1:1), remember there is always Source Aspect Ratio, Pixel Aspect Ratio and Display Aspect Ratio.
    • DB83
      • And just to throw another cog in the wheel, analogue tv transmissions are 625 lines (NTSC 525 lines).
    • pandy
      • Yes but it is not related to luminance bandwidth and sampling rate - there is 576 (480/486) visible lines only remain are so called VBI lines and are used to transmit vertical synchronization + equalization pulses and to transmit various types of data information (teletext, WSS, VPS, Closed Captions etc).
    • Cornucpia
      • One of the primary rules of video, which all here should know by now is: 
        Display Aspect Ratio = Pixel Aspect Ratio * Storage Aspect Ratio
      • The DAR = 4:3 = 1.33333 and the SAR = 1.25, as you have mentioned. So plugging those figures into the equation, 1.33333 = ? * 1.25, or rearranging it 1.33333 / 1.25 = ?. Solving it exactly gives: 1.06666666. This is quite close to the standard PAL PAR for non-widescreen: 59/54 or 1.0925.
      • The difference has to do with the fact that in sampling analog PAL signals, it is usually only ~702 of the 720 width that uses active pixels.
      • And 702/576 (or 1.21875) plugged into that original equation gives a PAR of ~1.094. And, since most devices like familiarity, the width of 704 is often used in Rec.601-compliant digital equivalent of PAL analog signals. 704/576 (1.22222) plugged into that equation gives a PAR of ~1.090909. Another standard ratio for PAL PAR non-widescreen is 12/11 or 1.090909. Look familiar?
      • As pandy and jagabo were mentioning, 768 is just the Square Pixel EQUIVALENT to 720's native non-square pixels.
      • Solving that same equation using square (1:1) PAR: 1.333333 = 1 * (? / 576) = 1.3333333 * 576 = ? ..... 768.
    • 2Bdecided
      • In the DVD and digital broadcast world, high quality "PAL" is 720x576, or 704x576 (i.e. with the parts that's not actually used in the analogue world removed - the extra 8 pixels either side were just included in the standard as a tolerance).
      • On quality compromised digital broadcasts it can be 544x576, 480x576, 352x576, and even 352x288 - just like 720vs704, some pixels are sometimes left off either side, making the horizontal pixel count even smaller (e.g. 528x576).
      • All of these resolutions can represent a 4x3 picture or x 16x9 picture.
      • 768x576 is only ever used when manipulating "PAL" video in systems that only understand square pixels. It's basically true to say that real "PAL" video never actually has square pixels.
      • Capture at 720x576. Crop to 704x576 if you want.
    • pandy
      • Question is a bit incorrect - PAL line length (visible part) is in fact equal 52.3us and it can have unlimited number of pixels (this depend only from sampling speed), for typical PAL B/G video signal bandwidth from practical perspective can't be bigger than 5.2MHz assuming fancy DSP involved - standard define bandwidth as 5MHz. Thus real resolution for PAL is (for 13.5MHz sampling) close to approx. 544 pixels.
      • For 13.5MHz maximum bandwidth is 6.75MHz, 1 pixel period is 1/13.5MHz=(approx) 74.074ns, if line period is equal to 52.3 us i.e. 52300ns then maximum number of pixels can be 52300/74.074=706.05, now for 5.2MHz PAL B/G standard bandwidth number of pixels is equal (5.2MHz/6.75MHz)*706.05=543.92 pixel.
      • This bandwidth limitation is usually common for non digital (analog RF broadcast) sources.
  • 720x576 vs 702x576, PAR confusion? - digitalFAQ Forum
    • I need to settle this once and for all. The proper PAR and SAR for PAL SD material.
    • The 720x576 SAR is the standard for both 16:9 and 4:3 material.
    • But the problem starts when PAR and nominal analogue blanking come into place.
    • The bottom line is, should 720x576 DV be displayed at 1,067 or 1,094, and therefore should I master to 720 with 1,067, or 1,094 with blanking bars at sides?
    • To conclude the 1,094 PAR is the proper one for PAL and VLC is not displaying PAL image properly, but the video itself is carrying 4:3 flag, which is correct for active image, but not taking blanking/overscan into account.
  • How do i upscale PAL - VideoHelp Forum
    • It can be confusing. If you have mpeg2, a dvd source or DV.avi then you have 720x576 which have a 4:3 DAR flag to display that 720x576 as 768x576 which is now perfect 4:3. Other 720x576 sources will report as 5:4
    • If you must upscale you an choose practically any size you desire. But there are caveats. Your source is interlaced and if you crop anything you must de-interlace before resizing.
    • Yes. 1280x960 is valid 4:3, but so is 1440*1080 which can save any further scaling in your player/tv.
    • Yes as hello_hello mentioned your VHS footage is only about 704x576, De-interlace first, crop to 704x576 and resize to 1440x1080 for a perfect 4:3 square pixel, resizing from 720x576 will not give you an accurate 4:3 aspect ratio.
    • DVD has actually a legal resolution of 704x576 for PAL/SECAM and 704x480 for all NTSC variants, I haven't seen any mention to 702 being an official standard, But in practice the junk surrounding the frame is never the exact number, it varies from tape format to another and from a standard to another, But when I crop I always base my calculations on 704. It seems to be the most accurate, I even did a circle test in one of the threads to demonstrate it a while back.
    • Lots more maths and discussion here.
  • Blackmagic Forum • View topic - MP4 PAL 720x576 4:3 pixel ratio export in DVR
    • 59:54 is correct as long as you have clap atom (clean aperture) set to 9. This means you have active area of 720-9 pixels for each side, so 702. Then when you calculate you need 59:54 PAR to get DAR as 4:3.
    • If you don't have clap set then you use "digital" flagging as 16:15 (720/576*16/15=1.333)
    • This might be useful in the future.
  • Why is NTSC showing 720 x 540, and not 480? - Moho Forum
    • Has a table of converting resolutions between : Rectangular Size --> Square Size
  • VHS conversion resolution? - digitalFAQ Forum
    • Q:
      • Greetings, I have read many articles on the topic of what resolution to capture VHS tapes in, but all the information just makes my head spin.
      • I would like to get a definitive answer on, in digital terms, what resolution the modes of VHS and S-VHS would be in, and if PAL or NTSC will affect those resolutions. SP, LP, SLP/EP, for both VHS and S-VHS to be clear.
    • A:
      • For all PAL VHS captures, regardless of the tape speed (SP, LP...), the normal resolution is 720x576, to be displayed at 4:3. Rarely, some gear will capture at 768x576, but it is uncommon.
      • VHS, S-VHS, SP, LP, SLP you name it, All SD analog video tape formats are captured at 720x576 for PAL/SECAM and 720x480 for NTSC, That's the native sampling rate per standard, only 704 out of 720 is actually contain the active image, So crop to 704x576 for PAL/SECAM and 704x480 for NTSC then set your aspect ratio to 4:3 during encoding and everything will work out just fine.
      • Sound Issue: Noise floor is the basic level of noise and hiss in a system that is always there whether or not there is a recorded signal. It comes from the electronics, the tape, the electromagnetic signals in the air around the gear, and so on. The signal to noise ratio you see in specs typically compares the desired signal level to the noise floor.
      • Read the forum for very techicnal discussion and futher explanation.
  • CGTalk | video sizes/aspect ratio - the answer!
    • okay, here is the definitive answer of what size video and aspect ratio you should use straight from the horses mouth (i.e. me). if you follow these guidelines you will not go wrong ever!
    • There are some different PAR values here.
  • Aspect Ratio and Digital Video | miraizon
    • This page discusses how aspect ratio works in digital video and common problems associated with editing and playback of anamorphic video.
    • Anamorphic Frame Size
    • Display Aspect Ratio
    • Square Pixel Frame Size
  • Aspect ratios | Doom9.net - A DVD video stream is 720x480, right? But 720/480 = 1.5 which is an impossible aspect ratio for a movie. And what about full screen, widescreen, anamorphic, etc? Many people are unfamiliar with these terms and are unsure about how to resize. This article tries to explain some of these mysteries.
  • Can someone EXPLAIN the whole "720x480" thing to me? - VideoHelp Forum
    • My DV camera obviously captures its video in 720x480, and I'm just curious what the thought process was behind this whole idea. That is, why capturing a 4:3 video will end up as 720x480, which is obviously NOT 4:3 and has to be filtered to play correctly (or so it appears to uneducated me).
    • DV uses non-square pixels, and these are adjusted by your player on playback. NTSC DVD also uses 720 x 480. Just to add to the fun, 16:9 (widescreen) images are also 720 x 480 (NTSC) or 720 x 576 (PAL).
    • The choice of 720x480 had to do with early digital video for broadcast. 704 (with the frame padded to 720x480) pixels were deemed necessary to match the horizontal visual resolution of high quality studio analog video. 480 was chosen because it's the nearst mod16 size that can capture all the resolution of the 486 scan lines of NTSC video (6 are cropped away).
    • So is 720x480 a square-pixel representation of a 480i video?
      • No. Standard definition 480i is 4:3. 720:480 = 3:2. The pixels are 10 percent taller than they are wide (PAR = 10:11). Note that a 720x480 video actually contains the 4:3 image in a 704x480 portion of the frame. There are 8 pixels added to each side for padding. So the full 720 pixel wide frame is slightly wider than 4:3. Using just the 704x480 part:
      • DAR = PAR * SAR
4:3 = 10:11 * 704:480
4/3 = (10/11) * (704/480)
4/3 = (10 * 704) / (11 * 480)
4/3 = 7040 / 5280
1.333 = 1.333

Troubleshooting

  • Video Artifacts
    • Time base correction - Wikipedia
    • Noise reduction - Wikipedia
    • Analog Artifacts - Browse by Tags | AVAA - A giant list of all possible artifacts with examples.
    • GitHub - joncampbell123/composite-video-simulator
      • Code to process video to simulate analog composite video.
      • Analog composite video simulation (for retro video-like video production).
      • The reason for this project is to provide the internet a better simulation of composite video-based emulation, especially for the rash of people on YouTube who all have their own ideas on what VHS artifacts looks like.
    • Dot Crawl
      • Use S-Video and not Composite/RCA to reduce or remove this issue.
      • GitHub - zhuker/ntsc: NTSC video simulator
        • This is a python3.6 rewrite of https://github.com/joncampbell123/composite-video-simulator intended for use in analog artifact removal neural networks but can also be used for artistic purposes
        • The ultimate goal is to reproduce all of the artifacts described here https://bavc.github.io/avaa/tags.html#video
        • A composite video artifact, dot crawl occurs as a result of the multiplexing of luminance and chrominance information carried in the signal. Baseband NTSC signals carry these components as different frequencies, but when they are displayed together, the chroma information can be misinterpreted as luma. The result is the appearance of a moving line of beady dots. It is most apparent on horizontal borders between objects with high levels of saturation. Using a comb filter to process the video can reduce the distraction caused by dot crawl when migrating composite video sources, and the artifact may be eliminated through the use of s-video or component connections. However, if it is present in an original source transfer, it might be compounded in subsequent generations of composite video transfers.
        • Has some good examples of video artifacts.
      • Dot Crawl Artifacts from Composite Source? - VideoHelp Forum
        • Dot crawl is the result of incomplete separation of the chroma subcarrier and luma from a composite source. Basically, it's always a problem with composite sources -- the more saturated the colors the more dot crawl artifacts you get. Capture devices usually have 2d (spacial only) or 3d (spacial and temporal) filters to reduce dot crawl artifacts. The temporal component of these filters work well on still parts of the picture but not on moving parts (you risk ghosting if you apply it too strongly to moving parts of the picture) -- which is what you're seeing.
        • An easy way of reducing dot crawl is to blur it away. You can do this by downsizing to half width, then upscaling back to full width. This isn't acceptable with high quality video because the picture gets blurry. But VHS has such low resolution horizontally you can usually do this without harming the picture much. Try using VirtualDub's Resize filter in Lanczos3 mode to scale down to 360x480 then back to 720x480.
        • You can also use more sophisticated methods involving masks to limit the blur to edges, highly saturated areas, and moving areas. But I don't think it's necessary for this clip.
        • There are also dot craw filters for VirtualDub. But, as usual, they don't work really well on moving parts of the picture.
      • "Dot crawl" elimination help? - "Dot crawl" elimination help?
        • Dot crawl is a well know artifact in composite analog video.
        • It happens at highly contrasting colour edges and looks like an unstable checkerboard pattern.
        • It's caused by crosstalk between the luminance and chrominance signals. Depending on the direction of the interference it is also responsible for colour bleeding.
        • In the analog domain, usually some kind of comb filter is used to add some constructive interference to help minimise these issues.
        • If you are capturing yourself, try using component signals instead of composite signals, as that should get rid of these issues .
    • Horizontal Wobble
      • Horizontal wiggle and de-framing when capturing. Malfunctioning card? - VideoHelp Forum
        • An example video is on this post.
        • Hello there! Been searching the forum for a little while, but didn't find any problem exactly like mine. I'm trying to capture some Hi8 tapes from my childhood using a DigitNow! U170 capture card, and I've been experiencing an enormous amount of horizontal wiggle and de-framing, which didn't occur in playback, be it in the camera (HITACHI VM-E340E) or when connected to a TV.
        • A Time Base Corrector (TBC) is needed.
  • Audio Issues
    • VHS - Wikipedia
      • Hi-Fi audio is thus dependent on a much more exact alignment of the head switching point than is required for non-HiFi VHS machines. Misalignments may lead to imperfect joining of the signal, resulting in low-pitched buzzing. The problem is known as "head chatter", and tends to increase as the audio heads wear down.
    • VHS conversion resolution? - digitalFAQ Forum
      • Noise floor is the basic level of noise and hiss in a system that is always there whether or not there is a recorded signal. It comes from the electronics, the tape, the electromagnetic signals in the air around the gear, and so on. The signal to noise ratio you see in specs typically compares the desired signal level to the noise floor
  • VHS Specific
    • Audio Hiss on capture and playback of VHS capture
      • Remove audio hiss during VHS capture? - digitalFAQ Forum
        • There's an audio hiss in both playback and capture. Any thoughts on ways to remove this? I already tried switching from stereo to mono settings and while the hiss is less noticeable, it's still there.
        • In HiFi stereo there is no hiss, probably the VCR is just stays on mono linear track all the time since most low budget camcorder didn't record HiFi stereo anyway. One place to start is try to clean the fixed audio head with q-tips and alcohol.
          • This is possible. But sometimes you need to verify VCR settings, verify it is set to HiFi. Sometimes you'll find that only 1 channel is bad, so you'll capture only L or R HiFi channel.
        • All VHS has hiss to some degree, both linear and HiFi. Some decks do better that others, but it also depends on the tapes. I have mono tapes that hiss loudly in JVC, but not Panasonic. Some in Panasonic, not JVC. Some hiss regardless of deck.
        • Some other good information about the issue here.
    • Moldy VHS tapes cleaning tutorial (in 5 easy steps) - YouTube - The best and easiest way to clean you precious and rare VHS tapes and preserve them for years to come. This video tells you absolutely everything you need to know to remove mold once and for all, even from the nastiest tapes!

Standard Resolutions

In this section I will show you all of the resolutions you will come across and it can be used as a reference.

List of Resolutions and relevant information

This a list of various resolutions you will come across. There are others but you probably don't need those.

  • 1920x1080
    • 16:9
    • 1080p
  • 1440x1080
    • 4:3
    • 1080p
    • HDV
  • 1280x720
    • 16:9
    • 720p
  • 1024x576
    • 16:9
    • PAL DVD widescreen output (DAR)
  • 960x720
    • 4:3
    • 720p
  • 853x480
    • 16:9
    • NTSC DVD widescreen output (DAR)
  • 768x576
    • 4:3
    • PAL DVD square output (DAR)
  •  720x576
    • 5:4
    • 576i
    • PAL
      • Fat pixels
      • Interlaced
      • 25 Frames a second (fps)
      • 50 Fields a second
      • Storage aspect ratio (SAR): 5:4 (720×576)
      • Display aspect ratio (DAR): 4:3
      • Pixel aspect ratio (PAR): 59:54 or 1.093 (1.0925)
        • I have also seen 1.0940/1.094
      • All PAL videos are stored in this resolution on all medias.
  • 720x540
    • 4:3
    • NTSC and PAL effective display resolution (4:3)
    • There was never a proper widescreen format for VHS/PAL/NTSC analogue. There is only 4:3 in which a widescreen image is displayed as a rectangle within the square display, and the image is of lesser quality.
      DVDs are a different situation because they are natively digital. The player or TV would automatically crop the images or there would usually be a button on the TV remote to change to a 'Widescreen' display format.
    • NTSC DVD square output (DAR)
  • 720x480
    • 3:2
    • 480i
    • NTSC
      • Thin Pixels
      • Interlaced
      • 29.97 Frames a second (fps)
      • 59.94 Fields a second
      • Storage aspect ratio (SAR): 3:2 (720×480)
      • Display aspect ratio (DAR): 4:3
      • Pixel aspect ratio (PAR): 10:11 or 0.9 (0.909)
      • All NTSC videos are stored in this resolution on all medias.
  • 704x576
    • 11:9
    • PAL
  • 704x480
    • 22:15
    • NTSC
  • 352x576
    • 11:18
    • PAL
  • 352x480
    • 11:15
    • NTSC

 

PAL/NTSC Physical Media - Verified Values

I have just used random sources for these, some settings will always be the same and others will not.

  • PAL VHS
    • Video
      • Interlaced @ 25ps
      • Frame Resolution: 720x576
      • Field Resolution: 720x288
      • Format Output Resolution: 720x576
    • Audio
      • ?
  • PAL DVD
    • Video
      • Interlaced @ 25ps
      • Frame Resolution: 720x576
      • Field Resolution: 720x288
      • Format Output Resolution: 720x576
      • Bit rate mode: Variable
      • Bit rate: 5105 kb/s - 9800 kb/s
      • 4:3 DAR: 768x576
      • 16:9 DAR: 1024x576
    • Audio:
      • Format: AC-3 (Dolby Digital)
      • Bit rate mode: Constant
      • Bit rate: 192kb/s 
      • Sampling rate: 48Khz
    • MediaInfo
      • 16:9 
        General
Complete name                            : E:\VIDEO_TS\VTS_04_1.VOB
CompleteName_Last                        : E:\VIDEO_TS\VTS_04_3.VOB
Format                                   : MPEG-PS
File size                                : 2.10 GiB
Duration                                 : 55 min 35 s
Overall bit rate mode                    : Variable
Overall bit rate                         : 5 405 kb/s
Frame rate                               : 25.000 FPS

Video
ID                                       : 224 (0xE0)
Format                                   : MPEG Video
Format version                           : Version 2
Format profile                           : Main@Main
Format settings                          : CustomMatrix / BVOP
Format settings, BVOP                    : Yes
Format settings, Matrix                  : Custom
Format settings, GOP                     : Variable
Format settings, picture structure       : Frame
Duration                                 : 55 min 35 s
Bit rate mode                            : Variable
Bit rate                                 : 5 105 kb/s
Maximum bit rate                         : 9 800 kb/s
Width                                    : 720 pixels
Height                                   : 576 pixels
Display aspect ratio                     : 16:9
Frame rate                               : 25.000 FPS
Standard                                 : PAL
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Interlaced
Scan order                               : Top Field First
Compression mode                         : Lossy
Bits/(Pixel*Frame)                       : 0.492
Time code of first frame                 : 09:59:59:00
Time code source                         : Group of pictures header
GOP, Open/Closed                         : Open
GOP, Open/Closed of first frame          : Closed
Stream size                              : 1.98 GiB (94%)

Audio
ID                                       : 189 (0xBD)-128 (0x80)
Format                                   : AC-3
Format/Info                              : Audio Coding 3
Commercial name                          : Dolby Digital
Muxing mode                              : DVD-Video
Duration                                 : 55 min 35 s
Bit rate mode                            : Constant
Bit rate                                 : 192 kb/s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 31.250 FPS (1536 SPF)
Compression mode                         : Lossy
Stream size                              : 76.3 MiB (4%)
Service kind                             : Complete Main

Menu
Format                                   : DVD-Video
      • 4:3 
        General
Complete name                            : F:\VIDEO_TS\VTS_02_1.VOB
CompleteName_Last                        : F:\VIDEO_TS\VTS_02_2.VOB
Format                                   : MPEG-PS
File size                                : 1.72 GiB
Duration                                 : 6 s 720 ms
Overall bit rate mode                    : Variable
Overall bit rate                         : 2 196 Mb/s
Frame rate                               : 25.000 FPS

Video
ID                                       : 224 (0xE0)
Format                                   : MPEG Video
Format version                           : Version 2
Format profile                           : Main@Main
Format settings                          : CustomMatrix / BVOP
Format settings, BVOP                    : Yes
Format settings, Matrix                  : Custom
Format settings, GOP                     : M=3, N=12
Format settings, picture structure       : Frame
Duration                                 : 6 s 720 ms
Bit rate mode                            : Variable
Bit rate                                 : 2 152 Mb/s
Maximum bit rate                         : 7 000 kb/s
Width                                    : 720 pixels
Height                                   : 576 pixels
Display aspect ratio                     : 4:3
Frame rate                               : 25.000 FPS
Standard                                 : PAL
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Interlaced
Scan order                               : Top Field First
Compression mode                         : Lossy
Bits/(Pixel*Frame)                       : 207.557
Time code of first frame                 : 00:00:00:00
Time code source                         : Group of pictures header
GOP, Open/Closed                         : Closed
Stream size                              : 1.68 GiB (98%)

Audio
ID                                       : 189 (0xBD)-128 (0x80)
Format                                   : AC-3
Format/Info                              : Audio Coding 3
Commercial name                          : Dolby Digital
Muxing mode                              : DVD-Video
Duration                                 : 6 s 720 ms
Bit rate mode                            : Constant
Bit rate                                 : 192 kb/s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 31.250 FPS (1536 SPF)
Compression mode                         : Lossy
Stream size                              : 158 KiB (0%)
Service kind                             : Complete Main

Menu
Format                                   : DVD-Video
  • PAL DVD-RW (Home DVD recorder)
    • Video
      • Interlaced @ 25ps
      • Frame Resolution: 720x576
      • Field Resolution: 720x288
      • Format Output Resolution: 720x576
      • Bit rate mode: Constant
      • Bit rate: 9000 kb/s
      • 4:3 DAR: 768x576
      • 16:9 DAR: 1024x576
    • Audio:
      • Format: MPEG Audio
      • Bit rate mode: Constant
      • Bit rate: 384kb/s 
      • Sampling rate: 48.0kHz
    • MediaInfo 
      • General
Complete name                            : Z:\VIDEO_TS\VTS_01_1.VOB
CompleteName_Last                        : Z:\VIDEO_TS\VTS_01_5.VOB
Format                                   : MPEG-PS
File size                                : 4.18 GiB
Duration                                 : 1 h 2 min
Overall bit rate mode                    : Constant
Overall bit rate                         : 9 544 kb/s
Frame rate                               : 25.000 FPS

Video
ID                                       : 224 (0xE0)
Format                                   : MPEG Video
Format version                           : Version 2
Format profile                           : Main@Main
Format settings                          : CustomMatrix / BVOP
Format settings, BVOP                    : Yes
Format settings, Matrix                  : Custom
Format settings, GOP                     : M=3, N=12
Format settings, picture structure       : Frame
Duration                                 : 1 h 2 min
Bit rate mode                            : Constant
Bit rate                                 : 9 000 kb/s
Width                                    : 720 pixels
Height                                   : 576 pixels
Display aspect ratio                     : 4:3
Frame rate                               : 25.000 FPS
Standard                                 : PAL
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Interlaced
Scan order                               : Top Field First
Compression mode                         : Lossy
Bits/(Pixel*Frame)                       : 0.868
Time code of first frame                 : 00:00:00:00
Time code source                         : Group of pictures header
GOP, Open/Closed                         : Open
GOP, Open/Closed of first frame          : Closed
Stream size                              : 3.93 GiB (94%)

Audio
ID                                       : 192 (0xC0)
Format                                   : MPEG Audio
Format version                           : Version 1
Format profile                           : Layer 2
Duration                                 : 1 h 2 min
Bit rate mode                            : Constant
Bit rate                                 : 384 kb/s
Channel(s)                               : 2 channels
Sampling rate                            : 48.0 kHz
Frame rate                               : 41.667 FPS (1152 SPF)
Compression mode                         : Lossy
Stream size                              : 172 MiB (4%)

Menu
Format                                   : DVD-Video
  • PAL DV
    • Video
      • Interlaced @ 25ps
      • Frame Resolution: 720x576
      • Field Resolution: 720x288
      • Format Output Resolution: 720x576
      • Bit rate mode: Constant
      • Bit rate: 30Mb/s
      • 4:3 DAR: 768x576
      • 16:9 DAR: 1024x576
    • Audio:
      • Format: PCM
      • Bit rate mode: Constant
      • Bit rate: 1536kb/s 
      • Sampling rate: 48Khz
      • Bit depth: 16bits
    • MediaInfo
      • Tape 1 
        General
Complete name                            : E:\DV Camera\RAW DV Camera dumps\toddler (25-12-14)\vid.13-10-18_16-35.00.avi
Format                                   : AVI
Format/Info                              : Audio Video Interleave
Commercial name                          : DVCAM
Format settings                          : BitmapInfoHeader / WaveFormatEx
File size                                : 56.9 MiB
Duration                                 : 15 s 601 ms
Overall bit rate mode                    : Constant
Overall bit rate                         : 30.6 Mb/s
Frame rate                               : 25.000 FPS
Recorded date                            : 2013-10-18 16:35:56.000

Video
ID                                       : 0
Format                                   : DV
Commercial name                          : DVCAM
Codec ID                                 : dvsd
Codec ID/Hint                            : Sony
Duration                                 : 15 s 600 ms
Bit rate mode                            : Constant
Bit rate                                 : 24.4 Mb/s
Width                                    : 720 pixels
Height                                   : 576 pixels
Display aspect ratio                     : 4:3
Frame rate mode                          : Constant
Frame rate                               : 25.000 FPS
Standard                                 : PAL
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Interlaced
Scan order                               : Bottom Field First
Compression mode                         : Lossy
Bits/(Pixel*Frame)                       : 2.357
Time code of first frame                 : 00:07:38:20
Time code source                         : Subcode time code
Stream size                              : 53.6 MiB (94%)

Audio
ID                                       : 1
Format                                   : PCM
Format settings                          : Little / Signed
Codec ID                                 : 1
Duration                                 : 15 s 601 ms
Bit rate mode                            : Constant
Bit rate                                 : 1 536 kb/s
Channel(s)                               : 2 channels
Sampling rate                            : 48.0 kHz
Bit depth                                : 16 bits
Stream size                              : 2.86 MiB (5%)
Alignment                                : Aligned on interleaves
Interleave, duration                     : 40  ms (1.00 video frame)
Interleave, preload duration             : 40  ms
      • Tape 2 
        General
Complete name                            : E:\DV Camera\RAW DV Camera dumps\carnival cruise 2007 - vid.06-01-01_00-00.00.avi
Format                                   : AVI
Format/Info                              : Audio Video Interleave
Commercial name                          : DV
Format profile                           : OpenDML
Format settings                          : BitmapInfoHeader / WaveFormatEx
File size                                : 13.0 GiB
Duration                                 : 1 h 1 min
Overall bit rate mode                    : Constant
Overall bit rate                         : 30.0 Mb/s
Frame rate                               : 25.000 FPS
Recorded date                            : 2006-01-01 00:00:00.000
 
Video
ID                                       : 0
Format                                   : DV
Codec ID                                 : dvsd
Codec ID/Hint                            : Sony
Duration                                 : 1 h 1 min
Bit rate mode                            : Constant
Bit rate                                 : 24.4 Mb/s
Width                                    : 720 pixels
Height                                   : 576 pixels
Display aspect ratio                     : 4:3
Frame rate mode                          : Constant
Frame rate                               : 25.000 FPS
Standard                                 : PAL
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Interlaced
Scan order                               : Bottom Field First
Compression mode                         : Lossy
Bits/(Pixel*Frame)                       : 2.357
Time code of first frame                 : 00:24:59:01
Time code source                         : Subcode time code
Stream size                              : 12.4 GiB (96%)
 
Audio
ID                                       : 1
Format                                   : PCM
Format settings                          : Little / Signed
Codec ID                                 : 1
Duration                                 : 1 h 1 min
Bit rate mode                            : Constant
Bit rate                                 : 1 024 kb/s
Channel(s)                               : 2 channels
Sampling rate                            : 32.0 kHz
Bit depth                                : 16 bits
Stream size                              : 453 MiB (3%)
Alignment                                : Aligned on interleaves
Interleave, duration                     : 40  ms (1.00 video frame)
Interleave, preload duration             : 40  ms
  • NTSC VHS
    • Video
      • Interlaced @ 29.97
      • Frame Resolution: 720x480
      • Field Resolution: 720x240
      • Format Output Resolution: 720x480
      • 4:3 DAR: 720x540
      • 16:9 DAR: 853x480
    • Audio
      • ?
  • NTSC DVD
    • Video
      • Interlaced @ 29.97
      • Frame Resolution: 720x480
      • Field Resolution: 720x240
      • Format Output Resolution: 720x480
      • 4:3 DAR: 720x540
      • 16:9 DAR: 853x480
    • Audio
      • ?
    • MediaInfo
      • 4:3 
        General
Complete name                            : E:\VIDEO_TS\VTS_01_1.VOB
CompleteName_Last                        : E:\VIDEO_TS\VTS_01_8.VOB
Format                                   : MPEG-PS
File size                                : 7.06 GiB
Duration                                 : 2 h 23 min
Overall bit rate mode                    : Variable
Overall bit rate                         : 7 023 kb/s
Frame rate                               : 29.970 FPS

Video
ID                                       : 224 (0xE0)
Format                                   : MPEG Video
Format version                           : Version 2
Format profile                           : Main@Main
Format settings                          : CustomMatrix / BVOP
Format settings, BVOP                    : Yes
Format settings, Matrix                  : Custom
Format settings, GOP                     : Variable
Format settings, picture structure       : Frame
Duration                                 : 2 h 23 min
Bit rate mode                            : Variable
Bit rate                                 : 6 691 kb/s
Maximum bit rate                         : 8 700 kb/s
Width                                    : 720 pixels
Height                                   : 480 pixels
Display aspect ratio                     : 4:3
Frame rate                               : 29.970 (30000/1001) FPS
Standard                                 : NTSC
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Interlaced
Scan order                               : Top Field First
Compression mode                         : Lossy
Bits/(Pixel*Frame)                       : 0.646
Time code of first frame                 : 00:59:59;00
Time code source                         : Group of pictures header
GOP, Open/Closed                         : Open
GOP, Open/Closed of first frame          : Closed
Stream size                              : 6.72 GiB (95%)

Audio
ID                                       : 189 (0xBD)-128 (0x80)
Format                                   : AC-3
Format/Info                              : Audio Coding 3
Commercial name                          : Dolby Digital
Format settings                          : Dolby Surround
Muxing mode                              : DVD-Video
Duration                                 : 2 h 23 min
Bit rate mode                            : Constant
Bit rate                                 : 192 kb/s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 31.250 FPS (1536 SPF)
Compression mode                         : Lossy
Stream size                              : 198 MiB (3%)
Service kind                             : Complete Main

Text
ID                                       : 224 (0xE0)-CC3
Format                                   : EIA-608
Muxing mode, more info                   : Muxed in Video #1
Duration                                 : 2 h 23 min
Start time (commands)                    : 200 ms
Start time                               : 701 ms
Bit rate mode                            : Constant
Stream size                              : 0.00 Byte (0%)
Count of frames before first event       : 15
Type of the first event                  : PopOn

Menu
Format                                   : DVD-Video
  • NTSC DVD-RW (Home DVD recorder)
    • Video
      • Interlaced @ 29.97
      • Frame Resolution: 720x480
      • Field Resolution: 720x240
      • Format Output Resolution: 720x480
      • 4:3 DAR: 720x540
      • 16:9 DAR: 853x480
    • Audio
      • ?
  • NTSC DV (guess)
    • Video
      • Interlaced @ 29.97
      • Frame Resolution: 720x480
      • Field Resolution: 720x240
      • Format Output Resolution: 720x480
      • 4:3 DAR: 720x540
      • 16:9 DAR: 853x480
    • Audio
      • ?

Notes

Research

A collections of my research links that dont fit into other categories

Useful Sites

  • OBS
    • Wiki - Wiki | OBS - If you're looking for any kind of assistance with OBS Studio, the site has a help portal with links to resources and our support channels.
  • VideoHelp
    • Homepage Video forums, video software downloads, guides, blu-ray players and media.
    • Software Downloads - Download free video and audio software. Old versions, user reviews, version history, screenshots.
    • Forum - This forum will help you with all your video and audio questions!
  • The Digital FAQ – Video, Photo, Web Hosting – Forum - Learn digital media and get video help, photo help, and web design help. Topics include capturing video, converting VHS to DVD, best blank DVDs, fixing DVD problems, digital photo tips, making web sites, and running web sites. High quality video services available. Forums, blogs, reviews, guides and articles.
  • Pricing | TapedMemories.com - This page has picture of all old storage media.

Capture Hardware

B-frames

  • Video compression picture types - Wikipedia
    • I-frames are the least compressible but don't require other video frames to decode.
    • P-frames can use data from previous frames to decompress and are more compressible than I-frames.
    • B-frames can use both previous and forward frames for data reference to get the highest amount of data compression.
  • B-Frames OBS - B-frame is short for bi-directional predictive frame, a form of video compression. In the 1800 frames of your one-minute video, you are the only moving object. The wall remains still and unchangeable. To cut down on the file size of your video, it is compressed. That is, only the pixels that change position from frame to frame are retained. B-frames perform compression by consulting the frames that come both before and after a frame. So if you have frames 1, 2, and 3, in order to render frame 2, a B-frame checks the pixel alignment on frames one and three. If the pixel alignment is different, then the changed pixels are the only ones that are stored on frame two and later rendered.
  • Help with the impact of raising Max B-frames | Reddit
  • keyframe interval and max b-frames for high FPS recordings | OBS Forums
    • The two parameters deal with quality. They trade off space for quality. If you record with a quality-based rate control such as CQP or CRF, you have infinite space, so you can just optimize for quality if you want. B-frames are the ones with the highest compression (most detail removed), so the more B-frames you insert, the lower the quality. So to optimize B frames for quality, you should use 0 B-frames (none at all) with CQP.
    • With key frames, it's the same, only on a higher level and the other way round. They contain a whole frame and are an anchor for P-frames, which have a higher higher compression (less detail removed) than the keyframes (but lower than the B-frames). So if you want higher quality, use more keyframes, which can be achieved by using a smaller keyframe interval. It has the side effect that a video with more keyframes is better seekable. With lower keyframe interval, video size increases vastly.
    • With CBR rate control, the effect is reversed, since you limit the bitrate. To achieve the forced bitrate, the encoder removes as much detail as needed. If you don't use B-frames or use a lower keyframe interval, the bitrate is consumed completely by the bigger frames, so the general quality must be lowered, which is very visible. So don't do this (don't use CBR for recording).
    • With the Simple/Standard outputs the interval is set in seconds (not frames) so 1 or 2 max. 1 will insert a Keyframe every 240 frames, 2 every 480 frames. If you decide you want to insert a Keyframe more often, like every 1/2 second (120 Frames) or 1/4 second (60 Frames) you'll need to learn how to use the Custom FFMPEG Output.
  • NVIDIA NvEnc Guide | OBS Forums
    • Look-ahead: Checked. This allows the encoder to dynamically select the number of B-Frames, between 0 and the number of B-Frames you specify. B-frames are great because they increase image quality, but they consume a lot of your available bitrate, so they reduce quality on high motion content. Look-ahead enables the best of both worlds. This feature is CUDA accelerated; toggle this off if your GPU utilization is high to ensure a smooth stream.
    • Max B-Frames: Set to 4. If you uncheck the Look-ahead option, reduce this to 2 B-Frames.
  • Question / Help - What is the "b-frames"? (NVENC) | OBS Forums
    • The more B-Frames the higher the quality, generally speaking. Is this even possible to set in NVEnc? Didn't think it was.
    • First, when its constant bitrate video, smaller size = better quality. Second, when its a hardware encoder, the computational increase doesn't matter as long as the ASIC or whatever it is can keep up (doesn't drop frames).
    • For x264 (or any software H.264 implementation), just cranking up B-Frames is bad because there's usually better features to turn on for more benefit and/or less CPU cost. For a hardware encoder, that rule doesn't apply unless someone has measured it and found that it does.
    • Well, when i set my b-frames to "2", i drop like ~60% of frames, so it becomes 10fps instead of 30 for me. Have no idea at all how to use it properly, so i just don't use it.

Bitrate

  • General
  • Different Bitrate control protocols
    • VBR
      • Has 2 settings bitrate settings:
        • Target Bitrate
        • Max Bitrate
      • This is an old way of recording video.
    • CBR
      • This is only used for streaming now to allow the remote system to plan for a constant stream.
      • This is an old way of recording video.
      • It is used by DVD-RWs so they know how much space is left. 10,000kb/s is one full DVD (4.7GB)
    • CQP
      • Constant quality control rather that controlling the bitrate. This is the modern way to record video.
      • Constant Quality Number (toolitip in HandBrake)
        • The encoder targets a certain quality.
        • The scale used by each encoder is different.
        • x264's scale is logarithmic and lower values correspond to higher quality. So small decreases in value will result in progressively larger increases in the resulting file size. A value of 0 means lossless and will result in a file size that is larger than the source, unless the source was also lossless.
        • Suggested values are: 18 to 20 for standard definition sources and 20 to 23 for high defination sources.
        • FFMpeg's and Theora's scale is more linear. These encoders do not have a lossless mode.
    • CRF
      • Constant quality control rather that controlling the bitrate. This is the modern way to record video.
    • Using the right `Rate Control` in OBS for streaming or recording | by Andrew Whitehead | Mobcrush Blog - Don't know your CBRs from your CQPs? You will soon!
      • CBR (Constant Bitrate)
      • ABR (Adaptive Bitrate)
      • CQP (Constant Quantization Parameter)
      • VBR (Variable Bitrate)
      • CRF (Constant Rate Factor)
      • Lossless
      • Let’s keep this simple. If you’re streaming, use CBR as every platform recommends it and it’s a reliable form of Rate Control. If you’re recording and need to be high quality, use CQP if the file size is no issue, or VBR if you want to keep file size more reasonable.
    • CBR or CQP :: OBS Studio General
      • An excellent explanation of the two.
      • CQP is a rate control method that keeps the quantization parameter constant throughout the encoding process. The quantization parameter controls the amount of compression applied to each frame, with higher values resulting in more compression and lower quality, and lower values resulting in less compression and higher quality. With CQP, the encoder maintains a constant level of compression, which can result in a consistent level of video quality, but at the cost of using varying amounts of bits for each frame.
      • CBR, on the other hand, keeps the bitrate of the encoded video stream constant throughout the encoding process, regardless of the complexity of the scene. This can result in a consistent level of video quality, but at the cost of potentially wasting bits on simpler scenes, as the same amount of bits are allocated to every frame.
    • In practical / video quality terms, what's the difference between CQP or VBR and CBR? What situations would someone use CQP / VBR over CBR for local recording? | Reddit
      • Short answer:
        • Constant QP means you get predictable quality, but unpredictable bit rate; VBR means you get predictable bit rate, but unpredictable quality.
      • Longer answer:
        • No, CQP means Constant Quantization Parameter, and it's actually just a flat compression ratio without regard to bit rates. It usually yields consistent quality, but not.. "intentionally", if you will.
        • And no, average bit rates of only 50 Mbps are not excessive, especially for 1440p 60fps. Depending on what you record and how, bit rates fluctuate very wildly, especially in pre-production video formats like ProRes.
    • Constant Bitrate (CBR) vs Variable Bitrate (VBR) - Learn the differences between CBR and VBR for video streaming and discover which is best for your needs. Explore the pros and cons of each technology with Digital Samba!
    • What is Video Bitrate and How to Choose the Best Settings - Castr's Blog
      • Bitrate (or bit rate) is how much information your video sends out per second from your device to an online platform.
      • Some great charts for bitrate.
      • Stereo should be 384Kbps
  • Examples
    • 10000 kb/s is about 4.5gb and hour i.e. the size of a DVD , dvd are 25fps, so and double that gives you 20,000 = 9gb and hour
    • Twitch max bitrate is 8000
  • Calculators
  • Streaming Bitrates
    • Broadcasting Guidelines | Twitch Help Portal - Our guidelines are set up in a way to find the right balance between visual quality and playback quality, where both the broadcaster and the viewer can benefit from. Read the info below to help you choose the Encoding, Bitrate, Resolution, and Framerate settings that provide the right balance for the game you're playing, your internet speed, and your computer's hardware. Remember: it's always better to have a stable stream than to push for a higher video quality that might cause you to drop frames or test the limits of your internet connection.
    • YouTube recommended upload encoding settings - YouTube Help - These are recommended upload encoding settings for your videos on YouTube.

Why are the capture files so large in OBS?

  • OBS Recording Produced Massive File Size | OBS Forums
    • You are recording using CQP.
    • That means that the encoder will use as much or as little bitrate as is needed to maintain a given image quality level.
    • Recording a (mostly unchanging) desktop isn't going to need much bitrate.
    • Recording a (constantly moving) first or third person perspective game is going to need A LOT more. Especially if there is a lot of detail and foliage.
    • Entirely normal and expected. To reduce recorded filesizes, bump your CQP level up from 18 to 22 or so. The larger the number, the worse the image quality, but the smaller the file size. Most who are recording for video creation do not keep their high-quality master footage for long, or have devoted recording drives. Good quality in real-time takes space. You CAN then throw the footage through something like Handbrake to re-encode it more efficiently, once it's a dead-file recording.
  • Recording File TOO large | OBS Forums
    • CQP = 14 will produce large file sizes.
    • If you want smaller file sizes with CQP, then you need to lower the quality setting. A higher number will result in lower quality and a smaller file size. Alternatively if you need more precision over file size then consider using CBR, for example 50000 kbps = 6.10 MB/s. Therefore 10 seconds = a 61 MB file size...
    • I would recommend trying 21 - 23 as your CQP value, see if the quality/file size ratio is acceptable. If not play around.
  • Question / Help - Recorded size too big | OBS Forums
    • You chose a CRF value of 10, which will create really huge video files. Sane values are 15-25 (lower values mean better quality and bigger file size). Rules of thumb: an increase of 3 will halve the size. Values below 15-18 (actual value depends on source material) are not distinguishable from the original.
  • Tips on reducing file size when recording locally | Reddit
    • Use CQP or CRF (depending on which encoder you're using) rather than CBR. That will dynamically change the bitrate in the background depending on what's being shown on screen while also maintaining visual quality. One of those two should always be used for local recordings, anyway, using CBR is a waste of storage space if you're doing anything except streaming.
    • A CQP of 26 is good for most things... (the lower the number, the more bitrate it uses... 23 uses double the bitrate of 26, 29 uses half of 26...). Tune it accordingly, start from 26.
    • Rawr_Mom
      • as the other poster said, use CQP instead of CBR. 18 is generally considered visually lossless, 24 will produce smaller files and is a popular choice
      • If your CPU has enough overhead, record x264, CRF. The files are generally smaller than estimated equivalents on nvenc. CPU usage preset (faster, slow, etc) reduces file size further at the cost of CPU utilisation.
      • check with your client if HEVC / H265 encoded video are fine with them; you can record in H265 with the StreamFX plugin for significantly smaller file sizes.
      • if you have tons of space to temporarily spare, you could record at an excessive bit rate (like CQP12, or even Lossless in simple mode) and then re-encode with ffmpeg, which will actually produce files that are quite a bit smaller than recording with those same settings from the outset.
      • For reference: I record 1440/60 at NVENC H264 CQP 12 and then re-encode to Nvenc H265 / HEVC CQP 22, and for 1440p video it's at the point where youtube re-encoding is the bottleneck. I can only tell the difference - on a paused frame of a character running quickly past the screen in poor lighting, looking at her face - if I upscale that final video to 4k for extra youtube bitrate.
  • VHS Capture Size Massive? - VideoHelp Forum
    • 8 Mbps = 1 MByte per second ... simply crunch the numbers and 3 hr tape ~ 10.8 GB. If lowering the bitrate gives unacceptable results, your only other option is to switch to a better compression codec that will give a better result at lower rates.
    • DV is 13GB per hour. Uncompressed can be several times that. Be happy that 4GB per hour is giving you the quality you want.
    • When using MPEG-2, at 720x480, (720x576 in your part of the world) I use very similar file sizes to yours to capture VHS to get satisfactory results (to my eyes). You are on par. I would highly suggest not using anything less than 8mbps as you will get noticeable quality loss for most captures, especially more so if you're capturing live sports events (motion, interlacing, etc).
  • File size way to big. | OBS Forums
    • My settings seem ok, and I can record the video without a problem, but when I complete the recording, the 2-hour long 720p output file is over 12gb in size.
    • Don't record with CBR or VBR, use CQP instead.
      • CQP is a quality-based encoding target that uses as much or as little bitrate as is needed to maintain a given image quality level.
      • 22 is the normal 'good' point, 16 for 'visually lossless', and 12 is generally the lowest you'll want to go even if you plan to edit the video later (to cut down on re-encoding artifacts). The lower the number, the closer to 'lossless' video it gets. But below 16 the filesizes get ridiculously large very fast.
  • Should my file size be this large? How can I lower file size without losing lots of quality? | OBS Forums
    • Q:
      • The issue is that the 1080p 60fps file ended up being 56.1 GB, which is a lot of storage usage. It also caused my 2 hour edited file to also be larger than usual at 7 GB, which my internet connection struggles to upload to YouTube.
      • How can I use less storage for videos like this, but still have high quality gameplay recordings for YouTube? I have thought about switching to Simple mode and just choosing High Quality, but I wasn't sure if that was considerably lower quality, and I would have to stop using multiple audio tracks (which I could if I had to).
    • A:
      • This is something you can only work out with trial & error. If you increase the CQ value, you decrease the quality, thus decrease the file size. Adding 3 to whatever CQ value you have is about half the file size, reducing by 3 is about double the file size.
      • Make a bunch of recordings with different CQ values and judge which quality you accept.

Multipass Mode

  • This option controls how and if the encoder prescans a frame so it can better compress it.
    • Single Pass: This means no multipass mode. The frame will be encoded directly
    • Two Passes (Quarter Resolution): The frame will be scanned at quarter resolution to help calculate the compression for the next pass.
      • This is the OBS default.
    • Two Passes (Full Resolution): The frame will be scanned at full resolution to help calculate the compression for the next pass.
  • New OBS Settings for 28.1 Questions | Reddit
    • Multipass Mode: This one confuses me as this is a new option. It defaulted to Two Passes (Quarter Resolution), but I set it to Single Pass for better performance (if I'm understanding that correctly).
  • NVENC streaming Preset and Multipass Mode - what settings are correct for streaming? | OBS Forums
    • What are the correct settings for streaming? Is there anything we should be guided by when choosing these options?
    • Is it a big diference betwenn P5 Slow Good Quality and P7 Slowest Best Quality when it comes to computer load (usage) during the stream?

Colour Space (sRGB / Rec. 601 /Rec. 709 / .....)

  • How to Choose the Right Video Color Space - How do you choose the right video color space for your project? I want to take you through a few basic color spaces and their applications.
  • Rec. 601 - Wikipedia
  • Rec. 709 - Wikipedia
  • Question / Help - Colors (YUV full/partial and 601/709 | OBS Forums
    • Defaults are generally recommended if recording to prevent decoding issues (709/partial). If streaming, you should be able to use any of them. I prefer 709/full range.
    • Full range is WAAAAAY better
  • REC 601 vs. REC 709 - When do I use which? | AVS Forum
  • Question / Help - 709 vs 601 | OBS Forums
    • Actually, in my opinion you don't really need to test much. Depending on your output resolution you should choose the standard color profile for that resolution.
      • Standard definition: BT.601
      • High-definition (720p/1080p): BT.709
      • Ultra-high-definition (4K/8K): BT.2020 (not available in OBS)
    • Everything uploaded to YouTube will be converted to BT.709, so keep that in mind if you use OBS for that.
    • But last time I checked I remember that Firefox always displays BT.601. I think some other browsers have had this issue as well.
  • Color Gamut: Understanding Rec.709, DCI-P3, and Rec.2020 - For current projectors on the market there are three main color gamut standards: Rec.709 (also known as BT.709), DCI-P3, and Rec.2020 (also known as BT.2020).
  • High precision color spaces (including HDR) · obsproject/obs-studio Wiki · GitHub
  • Rec.709 vs Rec.709-A: Explained - Filmmaking Elements - In this article, we are explaining difference between Rec.709 and Rec.709-A. In the realm of digital imaging and color representation, standardization is key to ensuring consistency across various display devices and platforms.
  • Is 709 actually better quality than 601? | Reddit
    • Q:
      • Many video encoding softwares allow you to choose between YUV color spaces 601 and 709. 709 is often referred to as "HD", and 601 as "SD". But does 709 actually produce better color quality? I know there's a visual difference in the case of greyscale, but I have yet to find anything documenting a visible difference in color quality between the two.
    • A:
      • I don't think color spaces have anything to do with quality/resolution.
      • Yes and no. Rec.601 is an old standard that specifies specifies both resolution and color space. Rec.709 is the newer standard for HD video which specifies the HD resolutions, and also a newer color space. So using 601 color space doesn't directly hurt your resolution, but mixing a 601 resolution with a Rec.709 color space (or visa versa) would be pretty weird and nonstandard, and in many cases would be displayed wrong.
      • Rec.2020 (The UHD standard) does specify a much larger color gamut that is quite different from 709 or 601, but don't worry about that for the immediate future.
      • You should be working to Rec.709 if your end result will be displayed via broadcast, youtube, mobile, etc. All of this equipment/software expects 709 input. You'll get the most accurate/reliable result. You'll only ever need to use 601 in fringe cases. There's just no reason not to work in 709.
  • Color spaces - REC.709 vs. sRGB | Image Engineering - If you are in a hurry or just not interested in some background information, here is the essence for you – HDTV (Rec. 709) and sRGB share the same primary chromaticities, but they have different transfer functions.
  • What exactly is Rec.709? | Redshark - What exactly is Rec. 709?
  • What is Rec.709? Things You Must Know!! - YouTube | Waqas Qazi - We'll look at what Rec.709 is and why you should care to get familiar with it.
  • YUY2 or RGB for vhs capture? - VideoHelp Forum
    • Almost every capture device captures in YUY2 or a similar YUV 4:2:2 colorspace -- because this is closest to what is transmitted over an s-video or composite cable. If you request RGB they simply convert the YUY2 to RGB, wasting CPU cycles and disk space, and losing quality.
    • = use YUY2 for VHS capture

Colour Range / RGB Colour Range, Limited or Full?

  • Full vs Partial Color Ranges EXPLAINED for Streaming | OBS Forums - EposVox
    • A subject of understandable confusion when it comes to streaming and content creation - especially with game consoles - is RGB Color Range settings. This is one of those things that you may have frustrations with even if you don’t know what I’m talking about. If you’ve had overly-punchy and dark video captures, unsaturated or washed out captures, or just generally want to know what this setting is - this post is for you.
    • This refers to the maximum and minimum luminance values (or white/black levels) in a video signal.
    • Typically TVs and videos formatted for TV only use the Limited (or Partial, or “Legal”) range of 16-235. This means that any information above 235 is seen as white and any below 16 is seen as black.
    • H264 is generally optimized for this Limited/Partial mode.
    • PC monitors, however, typically operate in the Full range of 0-255.
    • In OBS, the setting appears in the Advanced tab of settings, where (in my opinion) it should always be left on Partial. There are some exceptions where Full is okay for recording (which we’ll mention later) but for streaming and most general uses, this should be left on Partial.
    • = leave on Limited
  • OBS STUDIO: Full vs Partial Color Ranges EXPLAINED (Limited vs Legal) Streaming RGB Range StreamLabs - YouTube | EposVox
    • Today we're tackling a technical subject I get asked about all too often: RGB Color Range in OBS Studio, StreamLabs OBS, etc. This has to do with the available luminance values within an 8-bit video signal. I break down the differences between Full and Partial/Limited Range, which you should really be using, and when there are exceptions to this rule.
    • Limited/Partial colour range was called legal range.
    • You should rarely be needing to use full.
    • = leave on Limited
  • All Versions - Full vs Partial Color Ranges EXPLAINED for Streaming | OBS Forums - EposVox -
  • ColourSpace | Data vs. TV Levels - There are two fundamental basics to image levels - creative/grading systems that will output either Data range images (0-255 or 0-1023), or TV Legal levels (16-235 or 64-940), and displays that expect the input signal to be either Data range images, or TV Legal levels, and will display accordingly.

Capture Settings

  • Essay - Video Resolution - In the following article I would like to give you the tools necessary not only to understand our current NTSC video system, but also gain the ability to intelligently approach the new and upcoming video formats.
  • A Quick Guide to Digital Video Resolution and Aspect Ratio Conversions | WayBackMachine
    • Digital video resolution and aspect ratio conversions are more complicated than people generally think. This document tries to shed some light on these issues.
    • Has a conversion table.
  • Hi8/VHS to DVD: which bitrate do you recommend? - VideoHelp Forum
    • Half D1 (352x576 PAL or 352x480 NTSC) with 2-pass vbr and an average bitrate around 3000 kbit/s seems OK for VHS source.
      • This guy was right on the money.. I've done 100's of VHS movie conversions to DVD so far. You will NOT get any added quality going above 352x480 @ 3000 bitrate on VHS.
      • The tradeoff when doing it this way is, of course, you can usually take 2 nearly full VHS tapes. Encode with the above settings. And they will both fit on a single DVD-R.
    • Reilly - I've been doing it for a couple years, and trial and error have confirmed for me that your capture resolutions in vdub should be 352x480 or 360x480. You only need to use 720x480 for mini-DV. For laserdisc I tend to use 704x480. Here's how you tell.
      • Read the thread for the expanation.
  • Correct settings for capturing VHS - please help a newbie | OBS Forums
    • VHS tapes are aspect ratio 4:3, so there will always be black bars if you display this on a 16:9 monitor. You should record to a video file that most closely matches the source material, so record to a 4:3 aspect ratio file. The black bars are added by any media player at playback, but are not contained in the video file.
    • For VHS tapes, record to 768x576 (PAL) - depends on what the capture card is able to produce. In OBS, set Settings->Video->Base resolution and output resolution both to one of these resolutions. In Settings->Video set fps to 50 if you have PAL material or to 59.94 if you have NTSC material.
      Use simple output mode and set the recording quality to "High Quality" or "Indistinguishable Quality". The latter produces bigger files but the best quality.
    • Don't use any video filters with OBS. Record the material as closely to the original as possible, with all drops and damage present. Do any beautifying in a postprocessing step with video editing software. This way you can postprocess the same material over and over again until you are satisfied without the need to re-record from tape.
    • The first postprocessing step would probably be to deinterlace from 50 fps to 25 fps.
    • The next postprocessing steps would be to correct colors or cut unwanted stuff. Since the effective resolution with VHS is only half of the original video (384x288), you might also downscale to this or to a multiple of your recording resolution - this is something you need to work out with trial and error. This downscaling will lessen artifacts/noise created by bloating up the small VHS resolution to 768x576. When upscaled again to your monitor resolution by your media player, the video will look better.
    • There are many different resolution variants (see https://en.wikipedia.org/wiki/Standard-definition_television for example), so you might try variants before fully recording hours of material.
    • here are many details to consider if you want perfect conversion, for example the effective pixel aspect ratio of a VHS recording is not 4:3 - pixels not quadratic. OBS, on the other side, works with quadratic pixels only, so I recommended to record to 768x576. This is aspect ratio 4:3 with pixel aspect ratio 1:1. Actually one should record to 720x576 or 704x480, but this will result in some slightly off aspect ratio, so it's probably better to record 768 horizontally. You can ignore all this and never see any difference during recording and postprocessing. But it may be that after recording and postprocessing, if you actually watch your videos, you might observe circles are ovals actually and wonder why this is.
  • Which is the best resolution and bitrate for capturing VHS t - Alludo USER to USER Web Board
    • When you convert it to MPEG, you generally want to use the highest bitrate possible that still allows your program to fit on a DVD.
    • The maximum bitrate for a DVD is about 10,000kbps (audio & video combined). Most people recommend keeping it down to 8,000 for "burned" DVDs, because some players have trouble with high bitrates on burned DVDs.
    • At 6,000kbps you can get 90 minutes of good quality audio and Dolby audio. (A lot of commercial DVDs seem to be recorded around 6000.) You have to use a lower bitrate to get 90 minutes with LPCM audio. When I've squeezed more than 2 hours of video on a DVD, I really start to nitice the quality-loss.
    • All my captures for VHS transfer to DVD are:
      • FULL D1 720x480 (ntsc), 720x576(pal).
      • Variable bit rate 7000 - 8000 ( I also use Constant Bit Rate alot)
      • Mpeg or Dolby Audio
  • Resolution For NTSC VHS Video Tape | OBS Forums
    • What base & output resolutions should I use to capture old NTSC VHS video tape?
    • A clear question, but somewhat difficult to answer and to understand, because the pixel aspect ratio is not 1:1 as in today's digital video processing. Pixel aspect ratio not 1:1 means a pixel is not a quadrat but actually a rectangle.
    • According to https://en.wikipedia.org/wiki/Standard-definition_television, you should start with 704x480 as resolution in your capture device. It might also be necessary to use 720x480 instead of 704x480, if you get a "full frame" from the digitizer.
    • This should be rescaled within OBS to 640x480 (or 654x480 resp.) to make the pixel aspect ratio quadratic. To achieve this, right-click your source->Transform->Edit transform and set the scaling options like this:
  • Image format/resolutions of recordings in VHS format? - digitalFAQ Forum
    • sanlyn
      • Capturing PAL VHS at 720x576 or NTSC at 720x480 is considered the best size and aspect ratio compromise for most restoration processing and encoding purposes. It is the frame size for standard definition DVD, BluRay, and AVCHD, and can be encoded for 4:3 and 16:9 DAR. After deinterlacing it can be resized to square-pixel sixes for anything you want. Otherwise, if you capture at 768x576 and want to make a DVD or SD-BluRay, you'll have to resize and take a quality hit. Resizing always has a cost. It's best to use resizing methods offered by Avisynth.
      • PAL at 768x576 square-pixel is really an oddball size that isn't usable except for personal players. It can't be used for DVD or BluRay. If you post it on the internet it will be resized to a more standard frame for a website's players.
      • Lossless and/or unencoded AVi files do not store imbedded aspect ratio display data. They will display at the physical frame size and are not resized for different aspect ratios by media players. After your 720x576 AVI is encoded to something like h.264 or MPEG, you can set the display aspect ratio to whatever is appropriate.
    • lordsmurf
      • Capture 720x576, period, nothing more to discuss on it.
      • You never convert to 768x576, you never do anything at that size. DAR translates rectangular pixels to that size for playback, or 720x540, but nothing is stored that way.
  • Captured PAL VHS - Outputting to DVD - What Resolution Should I use? - digitalFAQ Forum
    • 720x480. VHS is interlaced. So is NTSC DVD, and so is PAL DVD. Never resize video while it's interlaced. Deinterlace first, then resize, then re-interlace. NTSC DVD is 29.97fps, not 25fps. If the PAL DVD is movie-based, it could have been made in a number of ways. We'd need a short sample. There are ways to do that in Avisynth and some other free apps without screwing up frames and motion, but I don't think you'll fall in love with Premiere's results. Let us know how it turns out.
  • Digitizing video cassettes on storage media! - GP
    • The formats are saved in 1: 1 quality of VHS / S-VHS / VHS-C / Video8 / Digital8 / Hi8 / DVCAM / MiniDV recordings up to 720 x 576 pixels PAL (Europe) or 720x480 NTSC (America) format in MOV or MPEG4 format with codec H.264 with an audio quality of 48 kHz and 16 kbit/s.
    • It is not the resolution that is decisive, but the quality of the media and how it is digitized, which is very important to us.
    • Wrongly advertised by competitors, but not done in the right way: You cannot create FullHD or 4K quality from a VHS resolution of 720 x 576 pixels.
  • graphics - NTSC scan lines and vertical resolution - Retrocomputing Stack Exchange
    • Q:
      • From https://en.wikipedia.org/wiki/BBC_Micro "the height of the graphics display was reduced to 200 scan lines to suit NTSC TVs". But NTSC is supposed to have 241 visible scan lines per half frame. Why wouldn't you want to make the graphics display vertical resolution 240 instead of 200?
    • A:
      • While nominally 241 scan lines were visible in the sense they contained video information, all TV sets hid a varying amount of scan lines on top and bottom (and left and right) by overscan and by the bezel in front of the screen.
      • So with a vertical resolution of 240, on most TV sets parts at the top and bottom would not be seen. While this doesn't matter much for movies, it's not a good thing if you want to do text editing.
      • This is also the reason while basically all homecomputers and game consoles had some sort of border (which often could be colored) around the center part of the image that carried information: It was to make sure this central part would be visible on all TV sets.
  • capture of VHS using VirtualDub - output size - VideoHelp Forum
    • Hi. It is my understanding that camcorders from the 1990s recorded at 720 x 480, and that would be remain the same when copied onto a VHS tape.
    • Most tapes don't have the same black bars on the left and right especially the 8mm formats, It is always better to capture at the native sampling rate of the ADC chip which is 720 samples and crop or mask later if needed taking into consideration the AR is 704:480 not 720:480.
    • I agree, capturing full frame with overscan is more flexible.
    • But how do you know that the native sampling rate of ADC chip is 720 samples? VirtualDub simply presents all available modes that a particular ADC is capable of, depending on ADC I see different values. So if I select 640x480 from the dropdown I assume that the ADC captures at 640 pixels, not VirtualDub re-samples 720 into 640.
    • It's by design, Never heard of Rec.601 standard? All capture cards are designed on that same standard except some few modern chinese knockoff's that use PC resolutions.
    • Rec 601 defines the format of component digital video and the way analog-digital as well as 525/60 and 625/50 interoperate. I don't see the direct relation to how a particular ADC samples video. My point is that properties like frame rate, frame size, color subsampling, etc that a capturing program displays come from a predefined list that is provided by the ADC. For example, when you switch from 29.97 to 25, the ADC samples video at 25fps. I presume that similarly, when you switch from 720x480 to 640x480, the ADC samples at 640x480 - in hardware. I may be wrong, of course..
  • 720x480 widescreen pixel aspect ratio wrong? - II'm confused about why the Vegas project preset for "NTSC DV Widescreen" sets a pixel aspect ratio of 1:1.2121. DV is 720x480 pixels, and the DV widescreen aspect ratio is 16:9. If you do the math, you find that (16/9) / (720/480) = 1.18518... So where did 1.2121 come from?

Recording Settings

  • VHS to OBS | OBS Forums
    • If you don't rescale, bicubic and lanczos is not applied. And about bitrate: since you're recording, use a quality based rate control like CQP (if you use nvenc on a Nvidia GPU) or CRF (if you use x264) or ICQ (if you use Quicksync on a Intel iGPU). CBR/VBR is for streaming only.
    • Best would be to use simple output mode where you just choose the desired quality and don't have to think about numbers.

Encoders / Decoders / Codecs / Formats

  • General
  • Example Captures/Streams
  • Testing
  • OBS
    • NVIDIA Nvenc Obs Guide | GeForce News | NVIDIA - Configure OBS to get the most quality out of your stream.
      • Base (Canvas) Resolution: Set the resolution you normally play at. That is, your desktop resolution (if you play in borderless mode), or the game resolution you normally enter (if you play in full screen).
      • Output (Scaled) Resolution: Enter the resolution appropriate for your Upload Speed and Bitrate, as we discussed in the previous section.
    • Best OBS Encoders Ranked - X264 Vs NVENC Vs AVC | Streamer's Haven
      • Best OBS Encoders Ranked - 1: (New)NVENC 2: NVENC 3: X264 4: H264/AVC (Advanced Media Framework) - Here's why.
      • There are two types of encoders: Software / Hardware
      • Covers the differences and nvidia and AMD versions.
      • On the other hand, hardware encoding is accomplished using a purpose-built chip that does not need to be processed by the CPU before sending it on its way.
      • AVC/H.264 (AMD Advanced Media Framework) = my video card hardware
    • High Quality Recording (in OBS Studio) | Xaymar - Pushing 1 Pixel at a time - Ever since publishing the guide on how to achieve the best possible NVIDIA NVENC quality with FFmpeg 4.3.x and below, people repeatedly ask me what the best possible recording settings are. So today, as a Christmas present, let me answer this question to the best of my knowledge and help all of you achieve a quality you've never seen before.
    • High Quality Recordings with NVIDIA NVENC (in OBS Studio) | Xaymar - Pushing 1 Pixel at a time - This guide has been merged into the following guides: High Quality Recording (in OBS Studio) with H.264/AVC High Quality Recording (in OBS Studio) with H.265/HEVC High Quality Recording (in OBS Studio) with AV1 Back to the Guide
    • High Quality Streaming with NVIDIA® NVENC (in OBS Studio) | Xaymar - Pushing 1 Pixel at a time - Streaming with more than one PC has been the leader in H.264 encoding for years, but NVIDIAs Turing and Ampere generation has put a significant dent into that lead. The new generation of GPUs with the brand new encoder brought comparable quality x264 medium – if you can find a GPU that is. Let’s take a look at what’s needed to set up your stream for massively improved quality.
    • Audio/Video Formats Guide | OBS Knowledge Base - An overview of audio and video formats available in OBS Studio.
      • For high quality local recording one should use the best quality hardware encoder available (AV1 > HEVC > H.264) together with high-bitrate AAC or lossless audio (e.g. ALAC).
      • MKV is the default container and recommended for most use cases, as it can be easily remuxed into a more compatible format. However, fragmented MP4/MOV may be a good fit for most users who wish to simply upload their videos onto platforms such as YouTube or edit them in common software like Adobe Premiere or DaVinci Resolve.
    • Hardware Encoding | OBS Knowledge Base - Choosing a Hardware Encoder
      • Hardware encoders, as opposed to the included x264 software encoder, are generally recommended for best performance as they take the workload off the CPU and to a specialised component in the GPU that can perform video encoding more efficiently. Modern hardware encoders provide very good quality video with minimal performance impact.
      • However, earlier generation hardware encoders provide a lower-quality image. They offer minimal performance impact in exchange for a reduction in quality at the same bitrates as software encoding using the default preset of veryfast. As such, they can be a last resort if software encoding is not possible such as due to performance constraints.
    • Wiki - AMF Options | OBS
    • Low latency, high performance x264 options for for most streaming services (Youtube, Facebook,...) | OBS Forums
    • OBS H.265 Users! What encoding settings do you guys use? | Reddit
      • For Recording:
        • Rate Control - CQP CQ Level - 16 Keyframe Interval - 0s Preset - Quality Profile - Main GPU - 0 Max B Frames - 2
      • For streaming:
        • (Only option is h264) Video Bitrate - 6000Kbps Audio Bitrate - 320 Encoder Preset - Quality
      • B frames are a type of compressed frame between keyframes which are complete images. They are like partial data that tells you what changes between keyframes rather than encoding a complete image. They help compress the video to a smaller file size.
      • CQP should give you the optimal file size to quality ratio (depending on the preset number) while CBR will give you a constant bitrate. So if you pick a bitrate that is higher than you need to encoder good video at that resolution, then no matter what happens on screen, it will always be clear. CQP will use less data when there is less motion. It should auto adjust when there is more motion to prevent blurriness though. I'm not sure what is wrong with your CQP recordings, try lowering it to 15.
  • GPU Selection
    • The different GPU manufactures have their own separate encoders on modern GPUs:
      • Hardware (AMD, H.264) = AMD
      • Hardware (QSV, H.264) = Intel = Quick Sync Video
      • Hardware (NVENC, H.264) = Nvidia = Nvdia Encoding
      • Hardware (NVENC, HEVC) = Nvidia = Nvdia Encoding = H.265
    • Which NVIDIA graphic cards do support NVENC technology? – Elgato - NVENC is a technology used by NVIDIA that handles video hardware encoding. Many NVIDIA GPUs support this technology, among others some...
    • Video Encode and Decode GPU Support Matrix | NVIDIA Developer - Get the latest video encoding and decoding support information for all NVIDIA GPU products.
    • List of Nvidia graphics processing units - Wikipedia - This list contains general information about graphics processing units (GPUs) and video cards from Nvidia, based on official specifications.
    • NVIDIA NvEnc Guide | OBS Forums
      • The objective of this guide is to help you understand how to use the NVIDIA encoder, NVENC, in OBS. Note: we have simplified some of the concepts to make this guide accessible to a wider audience.
      • GeForce RTX GPUs have dedicated hardware encoders (NVENC), letting you capture and stream content without impacting GPU or CPU performance.  
      • GeForce RTX Capabilities per GPU generation:
        • GTX 10 Series: H.264 and HEVC
        • GTX 16 Series: H.264 and HEVC
        • RTX 20 & 30 Series: H.264 and HEVC, and AI powered effects
        • RTX 40 Series: H.264, HEVC, AV1 and AI powered effects
        • NVENC is NVIDIA’s encoder. It’s a physical section of our GPUs that is dedicated to encoding only. This means that your GPU can operate normally regardless of whether you use this region to stream or record. Other encoders, such as x264, use your CPU to encode, which takes resources away from other programs such as your game. Advanced codecs like AV1 are unable to run on consumer CPUs. This is why using NVENC allows you to play games at a higher framerate and avoid stuttering, giving you and your viewers a better experience.
        • NVIDIA has also worked closely with OBS to help optimize OBS Studio for NVIDIA GPUs, improving performance and enabling the latest and greatest features for quality.
        • One additional advantage of NVENC is that typically, the same version of NVENC is used per GPU generation. For example, a GeForce RTX 4090 and a GeForce RTX 4050 both have the same encoder quality
        • Recommends Lanczos + 60fps
      • Look-ahead: Checked. This allows the encoder to dynamically select the number of B-Frames, between 0 and the number of B-Frames you specify. B-frames are great because they increase image quality, but they consume a lot of your available bitrate, so they reduce quality on high motion content. Look-ahead enables the best of both worlds. This feature is CUDA accelerated; toggle this off if your GPU utilization is high to ensure a smooth stream.
      • Max B-Frames: Set to 4. If you uncheck the Look-ahead option, reduce this to 2 B-Frames.
      • Downscale Filter = Lanczos (Sharpend scaling, 36 samples)
  • NVidia Only

Other Video Capture Tutorials

  • The Best Easy Way to Capture Analog Video (it's a little weird) - YouTube | Technology Connections
    • Describes the process in general.
    • He uses a composite to HDMi Upscaler and then a HDMI capture device and finds this the best result.
    • Finally shows you what he does in Adobe Premier Pro CC
    • 60 frames a second gives a smoother video like when playing the video
  • How to convert VHS videotape to 60p digital video (2016) - YouTube
    • This video and its method have been replaced by the video I have based method 2 on.
    • This uses VirtualDub to capture and HandBrake to transcode.
    • Sound should be one of the following:
      • PCM 48000Hz, Stero 16-bit
      • PCM 44100Hz, Stero 16-bit
  • Analog Video Capture follow-up - YouTube | Technology Connextras
    • @470s - component/composite/svideo - S-Video can prevent DOT Crawl. Comb filter will remove DOT crawl.
    • @770s - DOT Crawl = grainy look.
  • The Ultimate Video Recording, Encoding and Streaming Guide - Unreal Aussies
    • Over the next few posts I’ll take you through the main technical points of recording, encoding and streaming video, in particular game footage. Most people can set up scenes and webcams with just a little patience, trial and error. But so many people out there don’t understand some of the basic, yet crucial concepts that go on under the hood.
    • If you’re reading this, you’ve undoubtedly heard of NVENC, Fraps, x264, DxTory, Shadowplay and a bunch of other technologies. In this guide, I’ll be focusing on what I think are the best, yet still pretty easy to use.
    • OBS, HandbBrake, AviDemux and a lot of other related subjects.
  • CAPTURE CARD DOCUMENTATION - Latency, Decode Modes, Formats, & MORE! | OBS Forums | EposVox
    • IN THIS RESOURCE: I will provide extensive documentation about the connection types, supported decode modes, supported resolutions, frame rates, passthrough, and input latency (to preview) of every capture card I have access to.
      • Intro/Overview
      • Decode Mode Support
      • Notes on RGB Color Space
      • Format Support
      • Notes on Scaler Support
      • Input Latency Testing
      • Notes on “Bitrate” support
      • Testing Methodology
      • Limitations & Future Improvement
      • How to submit capture cards for testing
    • Some buyers are looking for capture cards that provide specific decode modes to the user. These are color compression formats (not to be confused with data compression) that affect the bandwidth required by the video feed through the device, as well as the total image quality.
    • YUY2 - 4:2:2 color space, uncompressed data stream
      • This is the most common, and generally the target you want to aim for
      • Requires more bandwidth over USB/PCIe bus, but has minimal system resource load and latency
  • The ULTIMATE VHS Capture Guide - YouTube
    • Your family home videos are slowly deteriorating, it's always best to transfer them to a digital format, however a good amount of people often transfer their tapes in substandard quality. This video will hopefully show you the best method to transfer your tapes.
    • Uses VirtualDub for the capture software.
    • Why not to use 'VHS --> DVD' on a combi recorder @ 352s

Technical Videos (Misc)

  • Compatible Color: The Ultimate Three-For-One Special - YouTube | Technology Connections
    • RCA's attempt at creating a new color television standard that would be compatible with existing black and white TVs initially faced technical challenges. However, it was an obviously great idea from a backward compatibility standpoint, and the National Television Systems Committee latched onto this idea and helped to propel RCA's idea to the real world. This is that story.
    • This explains how Luminance and Chrominace all work together to make a TV picture.
  • Macrovision: The Copy Protection in VHS - YouTube | Technology Connections - Did you ever try to copy one VHS tape to another and find that it just, well, didn’t work? Macrovision was the clever creation of what is now TiVo that managed to confuse a VCR without causing too much distress to a TV. In this video, we find out what it is, how to spot it, and how it works (with a healthy dose of speculation).

Capture Test Results

Capture File Sizes (Downscale Filter)

Here I ran some samples on my setup to see what results I would get and in particular, file size.

Capture 1 - (852x480 @ 30fps, Variable Bitrate, High Quality: High Quality, Medium File Size, Bicubic)

# OBS Output Settings
Output Mode: Simple
Recording Quality: High Quality: High Quality, Medium File Size
Recording Format: Matroska Video (.mkv)
Video Encoder: Hardware (AMD, H.264)
Audio Encoder: AAC (Default)

# OBS Video Settings
Base (Canvas) Resolution: 1920x1080
Output (Scaled) Resolution: 852x480
Downscale Filter: Bicubic (Sharpened scaling, 16 samples)
Common FPS Values: 30

# MediaInfo
Overall bit rate: 9344 kb/s (this was Variable Bitrate)
Writing application: Lavf60.3.100
Writing library: Lavf60.3.100
First video stream: 852x480 (4:3), at 30.000 FPS, AVC (component)(High@4.2)(CABAC / 4 Ref Frames)
First audio stream: 48.0 kHz, 2 channels, AAC LC

# File Size
1 hour = 4.0GB

Capture 2 - (852x480 @ 30fps, Variable Bitrate, High Quality: High Quality, Medium File Size, Lanczos)

# OBS Output Settings
Output Mode: Simple
Recording Quality: High Quality: High Quality, Medium File Size
Recording Format: Matroska Video (.mkv)
Video Encoder: Hardware (AMD, H.264)
Audio Encoder: AAC (Default)

# OBS Video Settings
Base (Canvas) Resolution: 1920x1080
Output (Scaled) Resolution: 852x480
Downscale Filter: Lanczos (Sharpened scaling, 36 samples)
Common FPS Values: 30

# MediaInfo
Overall bit rate: 9260 kb/s (this was Variable Bitrate)
Writing application: Lavf60.3.100
Writing library: Lavf60.3.100
First video stream: 852x480 (16:9), at 30.000 FPS, AVC (component)(High@4.2)(CABAC / 4 Ref Frames)
First audio stream: 48.0 kHz, 2 channels, AAC LC

# File Size
1 hour = 4.0GB

Capture 3 - (1920x1080 @ 30fps, Variable Bitrate, High Quality: High Quality, Medium File Size)

# OBS Output Settings
Output Mode: Simple
Recording Quality: High Quality: High Quality, Medium File Size
Recording Format: Matroska Video (.mkv)
Video Encoder: Hardware (AMD, H.264)
Audio Encoder: AAC (Default)

# OBS Video Settings
Base (Canvas) Resolution: 1920x1080
Output (Scaled) Resolution: 1920x1080
Downscale Filter: [Resolutions match, no downscaling required]
Common FPS Values: 30

# MediaInfo
Overall bit rate: 15.0Mb/s (this was Variable Bitrate)
Writing application: Lavf60.3.100
Writing library: Lavf60.3.100
First video stream: 1920x1080 (16:9), at 30.000 FPS, AVC (component)(High@4.2)(CABAC / 4 Ref Frames)
First audio stream: 48.0 kHz, 2 channels, AAC LC

# File Size
1 hour = 6.3GB

What I found

  • Bicubic and Lanczos downscale filters had no affect on the size of the file.
  • OBS `Simple Mode` uses Variable Bitrate
  • 1920x1080 (2,073,600 pixels) uses 2.3GB extra an hour over the 852x480 (408,960 pixels). 1080p has 507% the amount of pixels than 480p, but the 1080p file is only 157.5% larger which means it is getting much better compression for the quality.
Capture File Sizes (Video Encoder Settings)
  • Advanced Settings (used below, unless mentioned otherwise)
    • NVidia NVENC (H.264)
    • 720x576 @ 50fps
    • Audio: 48kHz Stereo @ 192kb/s
  • Advanced: CQP

    • CQP Level - 30: 439,003Kb / 15mins * 60mins = 1,756,012Kb/hour (2GB per hour) = 488KB/s = 3904kbps
      General
Unique ID                                : 35575077448124118703731273079815880816 (0x1AC382BCC8D8B589450B495661974070)
Complete name                            : H:\OBS Captures\Video Test Captures\CQP 30 - 2024-01-03 16-47-27.mkv
Format                                   : Matroska
Format version                           : Version 4
File size                                : 429 MiB
Duration                                 : 15 min 14 s
Overall bit rate mode                    : Variable
Overall bit rate                         : 3 933 kb/s
Frame rate                               : 50.000 FPS
Writing application                      : Lavf60.3.100
Writing library                          : Lavf60.3.100
ErrorDetectionType                       : Per level 1

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L3.1
Format settings                          : CABAC / 2 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 2 frames
Codec ID                                 : V_MPEG4/ISO/AVC
Duration                                 : 15 min 14 s
Bit rate mode                            : Variable
Maximum bit rate                         : 11.2 Mb/s
Width                                    : 720 pixels
Height                                   : 576 pixels
Display aspect ratio                     : 5:4
Frame rate mode                          : Constant
Frame rate                               : 50.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Default                                  : No
Forced                                   : No
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709

Audio
ID                                       : 2
Format                                   : AAC LC
Format/Info                              : Advanced Audio Codec Low Complexity
Codec ID                                 : A_AAC-2
Duration                                 : 15 min 14 s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 46.875 FPS (1024 SPF)
Compression mode                         : Lossy
Title                                    : Track1
Default                                  : No
Forced                                   : No
    • CQP Level - 26: 1,145,042Kb / 15mins * 60mins = 4,580,168Kb/hour  (4.5GB per hour) = 1272KB/s = 10176kbps
      General
Unique ID                                : 306329523498121085214532735022649234206 (0xE674EB933E0CE25DB75095E63A22D31E)
Complete name                            : H:\OBS Captures\Video Test Captures\CQP 26 - 2024-01-11 17-01-38.mkv
Format                                   : Matroska
Format version                           : Version 4
File size                                : 1.09 GiB
Duration                                 : 15 min 0 s
Overall bit rate mode                    : Variable
Overall bit rate                         : 10.4 Mb/s
Frame rate                               : 50.000 FPS
Writing application                      : Lavf60.3.100
Writing library                          : Lavf60.3.100
ErrorDetectionType                       : Per level 1

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L3.1
Format settings                          : CABAC / 2 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 2 frames
Codec ID                                 : V_MPEG4/ISO/AVC
Duration                                 : 15 min 0 s
Bit rate mode                            : Variable
Maximum bit rate                         : 11.2 Mb/s
Width                                    : 720 pixels
Height                                   : 576 pixels
Display aspect ratio                     : 5:4
Frame rate mode                          : Constant
Frame rate                               : 50.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Default                                  : No
Forced                                   : No
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709

Audio
ID                                       : 2
Format                                   : AAC LC
Format/Info                              : Advanced Audio Codec Low Complexity
Codec ID                                 : A_AAC-2
Duration                                 : 15 min 0 s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 46.875 FPS (1024 SPF)
Compression mode                         : Lossy
Title                                    : Track1
Default                                  : No
Forced                                   : No
    • CQP Level - 25: 1,341,655Kb / 15mins * 60mins = 5,366,620Kb/hour  (5.5GB per hour) = 1490KB/s = 11920bps
      General
Unique ID                                : 276776621679352528233475097513479577539 (0xD0393D0526DC6C71F7BB116058A31FC3)
Complete name                            : H:\OBS Captures\Video Test Captures\CQP 25 - 2024-01-11 17-50-58.mkv
Format                                   : Matroska
Format version                           : Version 4
File size                                : 1.28 GiB
Duration                                 : 15 min 0 s
Overall bit rate mode                    : Variable
Overall bit rate                         : 12.2 Mb/s
Frame rate                               : 50.000 FPS
Writing application                      : Lavf60.3.100
Writing library                          : Lavf60.3.100
ErrorDetectionType                       : Per level 1

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L3.1
Format settings                          : CABAC / 2 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 2 frames
Codec ID                                 : V_MPEG4/ISO/AVC
Duration                                 : 15 min 0 s
Bit rate mode                            : Variable
Maximum bit rate                         : 11.2 Mb/s
Width                                    : 720 pixels
Height                                   : 576 pixels
Display aspect ratio                     : 5:4
Frame rate mode                          : Constant
Frame rate                               : 50.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Default                                  : No
Forced                                   : No
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709

Audio
ID                                       : 2
Format                                   : AAC LC
Format/Info                              : Advanced Audio Codec Low Complexity
Codec ID                                 : A_AAC-2
Duration                                 : 15 min 0 s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 46.875 FPS (1024 SPF)
Compression mode                         : Lossy
Title                                    : Track1
Default                                  : No
Forced                                   : No
    • CQP Level - 24: 1,524,916b / 15mins * 60mins = 6,099,664Kb/hour  (6GB per hour) = 1695KB/s = 13560kbps
      General
Unique ID                                : 181228007543681335105184658854276209449 (0x88573EA151230C1148E391CF02279B29)
Complete name                            : H:\OBS Captures\Video Test Captures\CQP 24 - 2024-01-11 16-21-10.mkv
Format                                   : Matroska
Format version                           : Version 4
File size                                : 1.45 GiB
Duration                                 : 15 min 0 s
Overall bit rate mode                    : Variable
Overall bit rate                         : 13.9 Mb/s
Frame rate                               : 50.000 FPS
Writing application                      : Lavf60.3.100
Writing library                          : Lavf60.3.100
ErrorDetectionType                       : Per level 1

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L3.1
Format settings                          : CABAC / 2 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 2 frames
Codec ID                                 : V_MPEG4/ISO/AVC
Duration                                 : 15 min 0 s
Bit rate mode                            : Variable
Maximum bit rate                         : 11.2 Mb/s
Width                                    : 720 pixels
Height                                   : 576 pixels
Display aspect ratio                     : 5:4
Frame rate mode                          : Constant
Frame rate                               : 50.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Default                                  : No
Forced                                   : No
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709

Audio
ID                                       : 2
Format                                   : AAC LC
Format/Info                              : Advanced Audio Codec Low Complexity
Codec ID                                 : A_AAC-2
Duration                                 : 15 min 0 s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 46.875 FPS (1024 SPF)
Compression mode                         : Lossy
Title                                    : Track1
Default                                  : No
Forced                                   : No
    • CQP Level - 23: 1,701,721Kb / 15mins * 60mins = 6,806,884Kb/hour (7GB per hour) = 1891KB/s = 15128kbps
      General
Unique ID                                : 175663100252395411654604105638675747627 (0x84277B8476EC58D323B4D375876C132B)
Complete name                            : H:\OBS Captures\CQP 23 - 2024-01-07 16-32-54.mkv
Format                                   : Matroska
Format version                           : Version 4
File size                                : 1.62 GiB
Duration                                 : 15 min 0 s
Overall bit rate mode                    : Variable
Overall bit rate                         : 15.5 Mb/s
Frame rate                               : 50.000 FPS
Writing application                      : Lavf60.3.100
Writing library                          : Lavf60.3.100
ErrorDetectionType                       : Per level 1

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L3.1
Format settings                          : CABAC / 2 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 2 frames
Codec ID                                 : V_MPEG4/ISO/AVC
Duration                                 : 15 min 0 s
Bit rate mode                            : Variable
Maximum bit rate                         : 11.2 Mb/s
Width                                    : 720 pixels
Height                                   : 576 pixels
Display aspect ratio                     : 5:4
Frame rate mode                          : Constant
Frame rate                               : 50.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Default                                  : No
Forced                                   : No
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709

Audio
ID                                       : 2
Format                                   : AAC LC
Format/Info                              : Advanced Audio Codec Low Complexity
Codec ID                                 : A_AAC-2
Duration                                 : 15 min 0 s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 46.875 FPS (1024 SPF)
Compression mode                         : Lossy
Title                                    : Track1
Default                                  : No
Forced                                   : No
    • CQP Level - 20: 2,210,115Kb / 16mins * 60mins = 8,287,931Kb/hour  (8GB per hour) = 2302KB/s = 18416kbps
      General
Unique ID                                : 37435882224530099396630557843151568443 (0x1C29E37F07BDCED4CF44DC1153F2BE3B)
Complete name                            : H:\OBS Captures\Video Test Captures\CQP 20 - 2024-01-03 16-27-00.mkv
Format                                   : Matroska
Format version                           : Version 4
File size                                : 2.11 GiB
Duration                                 : 15 min 41 s
Overall bit rate mode                    : Variable
Overall bit rate                         : 19.2 Mb/s
Frame rate                               : 50.000 FPS
Writing application                      : Lavf60.3.100
Writing library                          : Lavf60.3.100
ErrorDetectionType                       : Per level 1

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L3.1
Format settings                          : CABAC / 2 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 2 frames
Codec ID                                 : V_MPEG4/ISO/AVC
Duration                                 : 15 min 41 s
Bit rate mode                            : Variable
Maximum bit rate                         : 11.2 Mb/s
Width                                    : 720 pixels
Height                                   : 576 pixels
Display aspect ratio                     : 5:4
Frame rate mode                          : Constant
Frame rate                               : 50.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Default                                  : No
Forced                                   : No
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709

Audio
ID                                       : 2
Format                                   : AAC LC
Format/Info                              : Advanced Audio Codec Low Complexity
Codec ID                                 : A_AAC-2
Duration                                 : 15 min 41 s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 46.875 FPS (1024 SPF)
Compression mode                         : Lossy
Title                                    : Track1
Default                                  : No
Forced                                   : No
    • CQP Level - 15: 15,577,813Kb / 85min  * 60mins = 10,996,103Kb/hour (11GB per hour) = 3054KB/s = 24432kbps
      General
Unique ID                                : 96810718493985795077012779441069682963 (0x48D510F06B91E51465656E97F256F113)
Complete name                            : H:\OBS Captures\Video Test Captures\CQP 15 - 2024-01-03 13-12-20.mkv
Format                                   : Matroska
Format version                           : Version 4
File size                                : 14.9 GiB
Duration                                 : 1 h 25 min
Overall bit rate mode                    : Variable
Overall bit rate                         : 24.9 Mb/s
Frame rate                               : 50.000 FPS
Writing application                      : Lavf60.3.100
Writing library                          : Lavf60.3.100
ErrorDetectionType                       : Per level 1

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L3.1
Format settings                          : CABAC / 2 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 2 frames
Codec ID                                 : V_MPEG4/ISO/AVC
Duration                                 : 1 h 25 min
Bit rate mode                            : Variable
Maximum bit rate                         : 11.2 Mb/s
Width                                    : 720 pixels
Height                                   : 576 pixels
Display aspect ratio                     : 5:4
Frame rate mode                          : Constant
Frame rate                               : 50.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Default                                  : No
Forced                                   : No
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709

Audio
ID                                       : 2
Format                                   : AAC LC
Format/Info                              : Advanced Audio Codec Low Complexity
Codec ID                                 : A_AAC-2
Duration                                 : 1 h 25 min
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 46.875 FPS (1024 SPF)
Compression mode                         : Lossy
Title                                    : Track1
Default                                  : No
Forced                                   : No
  • Advanced: CBR
    • CBR - 10000: 1,122,373Kb / 15mins * 60mins = 4,489,492Kb/hour (4.5GB per hour) = 1247KB/s = 9976kbps
      General
Unique ID                                : 321769289710963806689845544926612635147 (0xF21282D2759BAAB76E56DD7E43453E0B)
Complete name                            : H:\OBS Captures\Video Test Captures\CBR 10000 - 2024-01-06 11-56-19.mkv
Format                                   : Matroska
Format version                           : Version 4
File size                                : 1.07 GiB
Duration                                 : 15 min 0 s
Overall bit rate                         : 10.2 Mb/s
Frame rate                               : 50.000 FPS
Writing application                      : Lavf60.3.100
Writing library                          : Lavf60.3.100
ErrorDetectionType                       : Per level 1

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L3.1
Format settings                          : CABAC / 2 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 2 frames
Codec ID                                 : V_MPEG4/ISO/AVC
Duration                                 : 15 min 0 s
Bit rate mode                            : Constant
Nominal bit rate                         : 10 000 kb/s
Width                                    : 720 pixels
Height                                   : 576 pixels
Display aspect ratio                     : 5:4
Frame rate mode                          : Constant
Frame rate                               : 50.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.482
Default                                  : No
Forced                                   : No
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709

Audio
ID                                       : 2
Format                                   : AAC LC
Format/Info                              : Advanced Audio Codec Low Complexity
Codec ID                                 : A_AAC-2
Duration                                 : 15 min 0 s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 46.875 FPS (1024 SPF)
Compression mode                         : Lossy
Title                                    : Track1
Default                                  : No
Forced                                   : No
    • CBR - 20000: 2,221,418Kb / 15mins * 60mins = 8,885,672Kb/hour (9.0GB per hour) = 2468KB/s = 19744kbps
      General
Unique ID                                : 129082125544208984555618154264060931626 (0x611C502679799BD0FF2DA637DA8AAE2A)
Complete name                            : H:\OBS Captures\Video Test Captures\CBR 20000 - 2024-01-06 12-36-10.mkv
Format                                   : Matroska
Format version                           : Version 4
File size                                : 2.12 GiB
Duration                                 : 15 min 0 s
Overall bit rate                         : 20.2 Mb/s
Frame rate                               : 50.000 FPS
Writing application                      : Lavf60.3.100
Writing library                          : Lavf60.3.100
ErrorDetectionType                       : Per level 1

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L3.2
Format settings                          : CABAC / 2 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 2 frames
Codec ID                                 : V_MPEG4/ISO/AVC
Duration                                 : 15 min 0 s
Bit rate mode                            : Constant
Nominal bit rate                         : 20.0 Mb/s
Width                                    : 720 pixels
Height                                   : 576 pixels
Display aspect ratio                     : 5:4
Frame rate mode                          : Constant
Frame rate                               : 50.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.965
Default                                  : No
Forced                                   : No
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709

Audio
ID                                       : 2
Format                                   : AAC LC
Format/Info                              : Advanced Audio Codec Low Complexity
Codec ID                                 : A_AAC-2
Duration                                 : 15 min 0 s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 46.875 FPS (1024 SPF)
Compression mode                         : Lossy
Title                                    : Track1
Default                                  : No
Forced                                   : No
    • CBR - 30000: 3,322,366Kb / 15mins * 60mins = 13,289,464Kb/hour (13.5GB per hour) = 3692KB/s = 29536kbps
      General
Unique ID                                : 160223536104569543433758845467909810106 (0x7889EE3BAAA5FB065A42019528B1E7BA)
Complete name                            : H:\OBS Captures\Video Test Captures\CBR 30000 - 2024-01-06 12-53-05.mkv
Format                                   : Matroska
Format version                           : Version 4
File size                                : 3.17 GiB
Duration                                 : 15 min 0 s
Overall bit rate                         : 30.2 Mb/s
Frame rate                               : 50.000 FPS
Writing application                      : Lavf60.3.100
Writing library                          : Lavf60.3.100
ErrorDetectionType                       : Per level 1

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4.1
Format settings                          : CABAC / 2 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 2 frames
Codec ID                                 : V_MPEG4/ISO/AVC
Duration                                 : 15 min 0 s
Bit rate mode                            : Constant
Nominal bit rate                         : 30.0 Mb/s
Width                                    : 720 pixels
Height                                   : 576 pixels
Display aspect ratio                     : 5:4
Frame rate mode                          : Constant
Frame rate                               : 50.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 1.447
Default                                  : No
Forced                                   : No
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709

Audio
ID                                       : 2
Format                                   : AAC LC
Format/Info                              : Advanced Audio Codec Low Complexity
Codec ID                                 : A_AAC-2
Duration                                 : 15 min 0 s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 46.875 FPS (1024 SPF)
Compression mode                         : Lossy
Title                                    : Track1
Default                                  : No
Forced                                   : No
  • Advanced: VBR
    • This is just an example/guess for the settings and should not be taken as 100% for imaging VHS cassettes. Try it out though if you want.
    • Target: 3500, Max Bitrate:10000 : 413,185Kb / 15mins * 60mins = 1,652,740Kb/hour (1.7GB per hour) = 459KB/s = 3672kbps
      General
Unique ID                                : 155158777199815012946350775578937540317 (0x74BA7E56F511B9FB042D3062F87C3EDD)
Complete name                            : H:\OBS Captures\VBR Advanced - Target 3500 - Max 10000 - 2024-01-11 12-52-14.mkv
Format                                   : Matroska
Format version                           : Version 4
File size                                : 404 MiB
Duration                                 : 15 min 0 s
Overall bit rate mode                    : Variable
Overall bit rate                         : 3 758 kb/s
Frame rate                               : 50.000 FPS
Writing application                      : Lavf60.3.100
Writing library                          : Lavf60.3.100
ErrorDetectionType                       : Per level 1

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L3.1
Format settings                          : CABAC / 2 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 2 frames
Codec ID                                 : V_MPEG4/ISO/AVC
Duration                                 : 15 min 0 s
Bit rate mode                            : Variable
Maximum bit rate                         : 10 000 kb/s
Width                                    : 720 pixels
Height                                   : 576 pixels
Display aspect ratio                     : 5:4
Frame rate mode                          : Constant
Frame rate                               : 50.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Default                                  : No
Forced                                   : No
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709

Audio
ID                                       : 2
Format                                   : AAC LC
Format/Info                              : Advanced Audio Codec Low Complexity
Codec ID                                 : A_AAC-2
Duration                                 : 15 min 0 s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 46.875 FPS (1024 SPF)
Compression mode                         : Lossy
Title                                    : Track1
Default                                  : No
Forced                                   : No
  • Simple (VBR): High Quality, Medium File size

    • 11,293,973Kb / 60mins * 60mins = 11,293,973KB/hour (11GB per hour) = 3137KB/s = 25096kbps
      General
Unique ID                                : 76419877045050482474305776784949979518 (0x397DEED61F47EAA4FF92A44839E9297E)
Complete name                            : H:\OBS Captures\spice daewoo 50fps 709 720x576.mkv
Format                                   : Matroska
Format version                           : Version 4
File size                                : 10.8 GiB
Duration                                 : 1 h 0 min
Overall bit rate mode                    : Variable
Overall bit rate                         : 25.6 Mb/s
Frame rate                               : 50.000 FPS
Writing application                      : Lavf60.3.100
Writing library                          : Lavf60.3.100
ErrorDetectionType                       : Per level 1

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L3.1
Format settings                          : CABAC / 2 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 2 frames
Codec ID                                 : V_MPEG4/ISO/AVC
Duration                                 : 1 h 0 min
Bit rate mode                            : Variable
Maximum bit rate                         : 11.2 Mb/s
Width                                    : 720 pixels
Height                                   : 576 pixels
Display aspect ratio                     : 5:4
Frame rate mode                          : Constant
Frame rate                               : 50.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Default                                  : No
Forced                                   : No
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709

Audio
ID                                       : 2
Format                                   : AAC LC
Format/Info                              : Advanced Audio Codec Low Complexity
Codec ID                                 : A_AAC-2
Duration                                 : 1 h 0 min
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 46.875 FPS (1024 SPF)
Compression mode                         : Lossy
Title                                    : simple_aac_recording0
Default                                  : No
Forced                                   : No
  • DVD-RW (HQ settings) (Legacy Physical Media) (Not captured by OBS - for reference only)
    • Video
      • MPEG Video
      • CBR: 9000kb/s
      • 720x576i@25fps
    • Audio
      • MPEG Audio
      • 48kHz
      • Bit rate: 384 kb/s
    • Overall
      • bit rate: 9544 kb/s
    • Results
      • 1,048,512KB / 15mins * 60mins = 4,194,048Kb/hour (4GB per hour @ 25fps) = 1165KB/s = 9320kbps
  • DV Video (Legacy Physical Media) (Not captured by OBS - for reference only)
    • 720x576i@25fps
    • CBR: 30.0 Mb/s
    • 13,691,352Kb / 62min  * 60mins = 13,249,695Kb/hour (13.25GB per hour @ 25fps) = 3680KB/s = 29440kbps
  • Random Video (H.265 / HEVC / High Efficiency Video Coding)
    • 3840x1920@23.976
    • 868,840Kb / 59min  * 60mins = 883,566Kb/hour (885GB per hour @ 29.97fps) = 245KB/s = 1960kbps
    • The quality is excellent with these settings
      General
Unique ID                                : 2127013115158872757609751600123456789 (0x199A5D7DCF170A15FAA041123456789)
Complete name                            : E:\Moby Dick.mkv
Format                                   : Matroska
Format version                           : Version 4
File size                                : 848 MiB
Duration                                 : 59 min 23 s
Overall bit rate                         : 1 997 kb/s
Frame rate                               : 23.976 FPS
Encoded date                             : 2023-07-04 22:27:39 UTC
Writing application                      : HandBrake 1.4.0 2021071800
Writing library                          : Lavf58.76.100
ErrorDetectionType                       : Per level 1

Video
ID                                       : 1
Format                                   : HEVC
Format/Info                              : High Efficiency Video Coding
Format profile                           : Main 10@L5@High
HDR format                               : SMPTE ST 2086, HDR10 compatible
Codec ID                                 : V_MPEGH/ISO/HEVC
Duration                                 : 59 min 23 s
Width                                    : 3 840 pixels
Height                                   : 1 920 pixels
Display aspect ratio                     : 2.000
Frame rate mode                          : Constant
Frame rate                               : 23.976 (24000/1001) FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 10 bits
Writing library                          : x265 3.5+1-f0c1022b6:[Windows][GCC 9.2.0][64 bit] 10bit
Encoding settings                        : cpuid=1049583 / frame-threads=16 / numa-pools=16,16 / wpp / no-pmode / no-pme / no-psnr / no-ssim / log-level=2 / input-csp=1 / input-res=3840x1920 / interlace=0 / total-frames=0 / level-idc=50 / high-tier=1 / uhd-bd=0 / ref=1 / no-allow-non-conformance / repeat-headers / annexb / no-aud / no-hrd / info / hash=0 / no-temporal-layers / open-gop / min-keyint=24 / keyint=240 / gop-lookahead=10 / bframes=0 / b-adapt=0 / no-b-pyramid / bframe-bias=0 / rc-lookahead=12 / lookahead-slices=0 / scenecut=90 / hist-scenecut=0 / radl=0 / no-splice / no-intra-refresh / ctu=32 / min-cu-size=32 / no-rect / no-amp / max-tu-size=32 / tu-inter-depth=3 / tu-intra-depth=3 / limit-tu=3 / rdoq-level=0 / dynamic-rd=0.00 / no-ssim-rd / signhide / no-tskip / nr-intra=500 / nr-inter=500 / no-constrained-intra / strong-intra-smoothing / max-merge=5 / limit-refs=2 / no-limit-modes / me=2 / subme=7 / merange=57 / temporal-mvp / no-frame-dup / no-hme / weightp / no-weightb / no-analyze-src-pics / no-deblock / no-sao / no-sao-non-deblock / rd=1 / selective-sao=0 / early-skip / no-rskip / no-fast-intra / no-tskip-fast / no-cu-lossless / no-b-intra / no-splitrd-skip / rdpenalty=0 / psy-rd=0.00 / psy-rdoq=0.00 / no-rd-refine / no-lossless / cbqpoffs=0 / crqpoffs=0 / rc=crf / crf=19.0 / qcomp=1.00 / qpstep=0 / stats-write=0 / stats-read=0 / vbv-maxrate=100000 / vbv-bufsize=100000 / vbv-init=0.9 / min-vbv-fullness=50.0 / max-vbv-fullness=80.0 / crf-max=0.0 / crf-min=0.0 / ipratio=1.00 / aq-mode=3 / aq-strength=0.50 / no-cutree / zone-count=0 / no-strict-cbr / qg-size=32 / no-rc-grain / qpmax=69 / qpmin=0 / no-const-vbv / sar=1 / overscan=0 / videoformat=5 / range=1 / colorprim=9 / transfer=16 / colormatrix=9 / chromaloc=0 / display-window=0 / master-display=G(34000,16000)B(13250,34500)R(7500,3000)WP(15635,16450)L(10000000,50) / cll=341,95 / min-luma=0 / max-luma=4000 / log2-max-poc-lsb=8 / vui-timing-info / vui-hrd-info / slices=1 / no-opt-qp-pps / no-opt-ref-list-length-pps / no-multi-pass-opt-rps / scenecut-bias=0.90 / hist-threshold=0.03 / no-opt-cu-delta-qp / no-aq-motion / hdr10 / hdr10-opt / no-dhdr10-opt / no-idr-recovery-sei / analysis-reuse-level=0 / analysis-save-reuse-level=0 / analysis-load-reuse-level=0 / scale-factor=0 / refine-intra=0 / refine-inter=0 / refine-mv=1 / refine-ctu-distortion=0 / no-limit-sao / ctu-info=0 / no-lowpass-dct / refine-analysis-type=0 / copy-pic=1 / max-ausize-factor=1.0 / no-dynamic-refine / no-single-sei / no-hevc-aq / no-svt / no-field / qp-adaptation-range=1.00 / scenecut-aware-qp=0conformance-window-offsets / right=0 / bottom=0 / decoder-max-rate=0 / no-vbv-live-multi-pass
Default                                  : Yes
Forced                                   : No
Color range                              : Limited
colour_range_Original                    : Full
Color primaries                          : BT.2020
Transfer characteristics                 : PQ
Matrix coefficients                      : BT.2020 non-constant
Mastering display color primaries        : Display P3
Mastering display luminance              : min: 0.0050 cd/m2, max: 1000 cd/m2
Maximum Content Light Level              : 341
MaxCLL_Original                          : 341 cd/m2
Maximum Frame-Average Light Level        : 95
MaxFALL_Original                         : 95 cd/m2

Audio #1
ID                                       : 2
Format                                   : AC-3
Format/Info                              : Audio Coding 3
Commercial name                          : Dolby Digital
Codec ID                                 : A_AC3
Duration                                 : 59 min 23 s
Bit rate mode                            : Constant
Bit rate                                 : 256 kb/s
Channel(s)                               : 6 channels
Channel layout                           : L R C LFE Ls Rs
Sampling rate                            : 48.0 kHz
Frame rate                               : 31.250 FPS (1536 SPF)
Compression mode                         : Lossy
Delay relative to video                  : -5 ms
Stream size                              : 109 MiB (13%)
Title                                    : Surround
Language                                 : English
Service kind                             : Complete Main
Default                                  : Yes
Forced                                   : No

Audio #2
ID                                       : 3
Format                                   : AAC LC SBR
Format/Info                              : Advanced Audio Codec Low Complexity with Spectral Band Replication
Commercial name                          : HE-AAC
Format settings                          : NBC
Codec ID                                 : A_AAC-5
Duration                                 : 59 min 23 s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 23.438 FPS (2048 SPF)
Compression mode                         : Lossy
Delay relative to video                  : -105 ms
Title                                    : Stereo
Language                                 : English
Default                                  : No
Forced                                   : No

Text #1
ID                                       : 4
Format                                   : ASS
Codec ID                                 : S_TEXT/ASS
Codec ID/Info                            : Advanced Sub Station Alpha
Duration                                 : 58 min 16 s
Compression mode                         : Lossless
Language                                 : English
Default                                  : No
Forced                                   : No

Text #2
ID                                       : 5
Format                                   : ASS
Codec ID                                 : S_TEXT/ASS
Codec ID/Info                            : Advanced Sub Station Alpha
Duration                                 : 58 min 16 s
Compression mode                         : Lossless
Title                                    : SDH
Language                                 : English
Default                                  : No
Forced                                   : No

Text #3
ID                                       : 6
Format                                   : ASS
Codec ID                                 : S_TEXT/ASS
Codec ID/Info                            : Advanced Sub Station Alpha
Duration                                 : 59 min 16 s
Compression mode                         : Lossless
Language                                 : Arabic
Default                                  : No
Forced                                   : No

Text #4
ID                                       : 7
Format                                   : ASS
Codec ID                                 : S_TEXT/ASS
Codec ID/Info                            : Advanced Sub Station Alpha
Duration                                 : 59 min 16 s
Compression mode                         : Lossless
Language                                 : Bulgarian
Default                                  : No
Forced                                   : No

Text #5
ID                                       : 8
Format                                   : ASS
Codec ID                                 : S_TEXT/ASS
Codec ID/Info                            : Advanced Sub Station Alpha
Duration                                 : 59 min 16 s
Compression mode                         : Lossless
Title                                    : Chinese (Simplified)
Language                                 : Chinese
Default                                  : No
Forced                                   : No

Text #6
ID                                       : 9
Format                                   : ASS
Codec ID                                 : S_TEXT/ASS
Codec ID/Info                            : Advanced Sub Station Alpha
Duration                                 : 59 min 16 s
Compression mode                         : Lossless
Title                                    : Chinese (Traditional)
Language                                 : Chinese
Default                                  : No
Forced                                   : No

What I found (so far)

  • CQP Level 15 = High Quality, Medium File size, and have the same bit rate.
  • CQP 23 = Good for capturing VHS
  • CQP
    • Is the modern protocol for recording media.
    • It brings the reduced file sizes because it only uses what data is required to meet a defined quality setting.
    • You define the quality of the recording and the protocol does the rest.
  • CBR @ 10000kb/s is the same as a DVD, almost. A DVD max rate is 10000kb/s including audio.
  • CBR rates are the same irrespective of the resolution they encode. So the larger the image the less the quality.
  • Twitch Max bitrate is 8000kb/s and people can do a 1920x1080 stream with no issues using h.264
  • A H.265/HEVC video with 3840x1920@23.976 has extremely high quality for 883,566Kb an hour (885Mb)

 

 

Published in Media
Read more...
Sunday, 10 September 2023 09:14

My RAM Notes

These are a collection of my notes on PC, Desktop and Server RAM.

  • Memory for my TrueNAS = Unbuffered ECC RAM (UDIMM)
  • General
  • Identify RAM Type
    • ECC RAM has an extra RAM chip, so instead of 8 matching chips there will be 9 matching chips. This chip is used to store parity data.
    • Buffered/Registered RAM will always be ECC and will have an extra chip for each memory chip. These extra chips reduced the load on the motherboards RAM controller and allows for many more DIMM slots.
    • DataMemorySystems.com - Frequently Asked Questions about RAM
      • Q: How to tell ECC, Parity memory from Non-ECC, Non-Parity memory?
      • A: If your system has ECC or parity memory the chips are evenly divisible by three. How do you know which one you have? One way is to look at the part numbers on the chips of your module. If each chip has the same part number, you have ECC. If one chip is different, you have parity.
  • Memory Timings
  • Misc
  • Buffered and Unbuffered RAM
  • ECC RAM
    • Linus was right. - ECC Memory Explained - YouTube | Linus Tech Tips
      • It’s possible to use ECC server RAM inside of your regular desktop computer at home, but is it something you SHOULD do?
      • AMD, although has not validated ECC on their consumer platforms, they have left the technology enabled allowing the choice for motherboard manufacturers as to whether they support it or not.
      • ECC adds stability at a small performance cost.
      • ECC = Error Correction Code
      • Can correct bit flips and notify the user of these errors.
      • UDIMM ECC modules (unbuffered) will work in any motherboard that supports their capacity and the DDR4 standard but the ECC chip will only be active if we choose a motherboard that explicitly supports ECC.
      • DDR5 has ECC built into the standard.
    • I LOVE Paywalls. Thanks Intel! - ECC Support on Alder Lake - YouTube | Linus Tech Tips
      • 12th Gen Intel (Alder Lake) supports ECC memory, but you're going to need a specific chipset to utilize it. A chipset only available on expensive workstation motherboards that lack other features you might want... So just how badly do you need Error Correction Code memory in the first place?
      • Like Intel, AMD say ECC is a workstation and server class feature that general consumers probably don't need. they only validate it on their professional products but AMD have not outright disabled the function on their consumer CPUs and chipsets. This allows theri motherboard providers to activate ECC if they choose to.
    • ECC Memory vs. DDR5 Built in Data Checking - Infographic - Competitors are calling DDR5's Built in Data Checking ECC memory - but it is not the same.  This infographic help customers understand the difference - and why they should look for Intel based workstations with ECC memory.
    • ecc - What and how to check when determining if a memory stick will be compatible with a particular server? - Server Fault - Some Questions and answers on ECC RAM.
    • What Is ECC Memory in RAM? A Basic Definition | Tom's Hardware - What’s the meaning of ECC memory? ECC memory in RAM explained.
  • DDR5 and built-in ECC (On-Die ECC)
    • The in-built ECC of DDR5 is not the same as normal ECC and is for all intensive purposes just allows manufacturers to increase RAM density.
    • Is DDR5 ECC memory? | CORSAIR:EXPLORER - Is DDR5 ECC memory? We take a look to find out.
    • What is DDR5? The PC's next-gen memory, explained | PCWorld
      • Is DDR5 more future proof? Is it faster? And what about DDR5's latency? We answer those questions and more.
      • DDR5 does indeed include ECC (or error correction control) that can detect multi-bit errors and correct single-bit errors. It is, however, not what you’re expecting if your workload already requires the technology.
      • With traditional ECC, error detection and control is performed at all levels, including the data that is transferred to the CPU. With DDR5, ECC is integrated into each actual RAM chip but once it leaves the chip and begins its journey along that long narrow wire to the CPU, there is no ECC performed, meaning errors induced along the way aren’t its problem.
    • DDR5 Memory Specification Released: Setting the Stage for DDR5-6400 And Beyond | Anandtech - an in-depth look at the DDR5 spec.
    • Why DDR5 does NOT have ECC (by default) - YouTube | TechTechPotato
      • DDR5, when it was announced, had a new feature called 'On-Die ECC'. Too many of the press, and even the DRAM company marketing materials misunderstood this important technology. It is not traditional ECC, and in fact won't do much if you really need an ECC system. Here's what it really does.
      • Also explains ECC.
      • Non-ECC is cheaper to make and give betters speeds.
    • DDR5 - Questions and answers | Crucial UK
      • Q: Is Crucial DDR5 Desktop Memory classified as ECC memory because it has the on-die ECC (ODECC) feature?
      • A: No. Crucial DDR5 Desktop Memory is non-ECC memory. The ECC as it pertains to RDIMMs, LRDIMMs, ECC UDIMMs, and ECC SODIMMs is a function that requires additional DRAM at the module level so that platforms such as servers and workstations can correct for errors on individual modules (DIMMs). On-die ECC (ODECC), however, is a feature of the DDR5 component specification and should not be confused with the module-level ECC feature. Crucial DDR5 Desktop Memory is built with DDR5 components that include ODECC, however these modules do not include the additional components necessary for system level ECC.
Published in Hardware
Read more...
Page 4 of 95