• Home

Items filtered by date: December 2014

Friday, 18 July 2025 13:13

PCIe Lane Diagnostics

How many PCIe lanes is your NVMe drive using in Windows

To check how many PCIe lanes your NVMe drive is using in Windows use the following apps to get the information.

CrystalDiskInfo

How to use:

  1. Install and run CrystalDiskInfo.
  2. Select your NVMe drive from the list at the top.
  3. Look for a line like: 
    Transfer Mode: PCIe 3.0 x4 | PCIe 3.0 x4
  4. The first value is active lanes.
  5. The second is maximum supported lanes by the drive.

HWiNFO

How to use:

  1. Launch HWiNFO, only Sensors are needed, but you can use full mode.
  2. Expand "Bus"
  3. Expand "PCI Bus"
  4. Select "PCI Express Root Port", and under the "PCI Express section" on the right:
    • Version is the PCIe Generation
    • Read the "Maximum Link Width" and "Current Link Width" to get the NVMe port's lane information.
  5. Under the "PCI Express Root Port" you will probably see the following
    • "PCI Express x4 Bus"
      • "Samsung Electronics Pascal S4LV008 PCIe 4.0 x4 NVMe 2.0 SSD Controller" (or your drive)
        • Read the "Maximum Link Width" and "Current Link Width" to get the NVMe drives lane information.

The drive information in this tool is just information read from the tool, it does not have current lane, just information read from the firmware.

Samsung Magician

How to use:

  1. Open Samsung Magician.
  2. Select your NVMe drive.
  3. Under the "Drive Details" or "Interface" section, it will show something like: 
    Interface: PCIe Gen 4.0 x4
    • This means it's using 4 lanes on Gen 4.

How can I check the number of PCIe lanes in Windows?

  • Use: HWiNFO
  • Count Your PCIe Lanes! Easy Guide for Any PC
    • Confused about PCIe lanes on your PC? Don't worry! This guide shows you simple methods to check how many lanes your system has (No software needed).
    • System Information Tool
      • On Windows, you can use the built-in System Information tool to find information about your motherboard and its PCIe lanes.
      • Press Win+R to open the Run dialog, type “msinfo32,” and hit Enter.
      • In the System Information window, navigate to “Components” > “PCI.” There, you will see a list of all PCIe devices in your system and their current link widths (corresponding to the number of lanes).
    • Third-Party Software
      • Some third-party software tools can provide detailed information about your computer’s hardware, including PCIe lanes.
      • CPU-Z and GPU-Z are popular tools that can provide comprehensive information about the motherboard and connected PCIe devices.
    • How can I check the number of PCIe lanes in Windows?
      • You can check the number of PCIe lanes in Windows using the device manager by following these steps:
        1. Right-click on the Start button and select “Device Manager.”
        2. In the Device Manager window, expand the “System Devices” category.
        3. Look for an entry named “PCI Express Root Port” or “PCI Express x16 Port” (the exact name may vary depending on your system).
        4. Double-click on the “PCI Express Root Port” entry to open its properties.
        5. Go to the “Advanced” tab, and you should see information about the number of “Current Link Width” or “Max Link Width.” This value represents the number of PCIe lanes currently used by the port.

General Notes

 

Published in General
Read more...
Friday, 18 July 2025 13:12

USB to ISO

USB to ISO

Published in General
Read more...
Friday, 18 July 2025 13:11

Disk Image Handling

Glossary

  • P2V
    • (Physical-to-Virtual) conversion
    • P2V converts a physical server's operating system, applications, and data into a virtual machine format (like VMDK).
  • V2V
    • (Virtual-to-Virtual) conversion
    • the process of migrating a virtual machine (VM) from one virtualization platform to another.
  • Fixed Disk / "Thick" Provisions
    • These disk images are a set size and are never changed. They are roughly the size of the disk defined in the image.
  • Dynamic Disk / Growable / "Thin" Provisioned / "Sparse" Provisioned
    • the disk image file size can change and is not directly related to the size of the defined virtual disk it holds.
  • OVA/OVF
    • These are not disk images, but a standard to allow moving of virtual machines between platforms.
    • Deploy and Export OVF and OVA Templates
      • You can export virtual machines, virtual appliances, and vApps in Open Virtual Format (OVF) and Open Virtual Appliance (OVA). You can then deploy the OVF or OVA template in the same environment or in a different environment.
    • How To Convert Virtual Machines Between VirtualBox and VMware
      • Migrating to another virtual machine program can be intimidating.
      • If you use an ova extension, then all of the files will be combined into one Open Virtualization Format Archive.
      • If you use an ovf extensions, several files will be written separately.
  • Super Floppy
    • This is the term given to a type of USB formating where it has the following features:
      • No MBR
      • No Partition Table
      • 1 Partition that takes the whole of the drive
    • The term "super floppy" can be a bit ambiguous, but generally, when people refer to a "super floppy" in the context of USB drives, they often mean a removable drive (like a USB flash drive) that does not have partitions, behaving more like a traditional floppy disk. Some older disk management utilities, like Acronis Disk Director, might label such drives as "Super Floppy."
  • Boot Sector
  • Master Boot Record (MBR)
    • The MBR is physically the first sector of a data medium (e.g. hard drive or USB stick), which is used by computers for the booting, or start-up process. The computer must be equipped with a BIOS and a x86 operating system.
    • The searches for the active partition and then load the VBR
    • The MBR code is OS specific.
    • This is placed in the boot sector of a drive
    • Volume boot record - Wikipedia
      • A master boot record (MBR) is a type of boot sector in the first block of partitioned computer mass storage devices like fixed disks or removable drives intended for use with IBM PC-compatible systems and beyond. The concept of MBRs was publicly introduced in 1983 with PC DOS 2.0.
      • The MBR holds the information on how the disc's sectors (A.K.A. "blocks") are divided into partitions, each partition notionally containing a file system.
      • The MBR also contains executable code to function as a loader for the installed operating system—usually by passing control over to the loader's second stage, or in conjunction with each partition's volume boot record (VBR). This MBR code is usually referred to as a boot loader.
    • MBR (Master Boot Record) explained - IONOS UK - The master boot record is a vital piece of code for many PCs to start up. But how does it work, how is it structured and how can it be repaired if necessary?
  • Volume Boot Record (VBR)
    • This is where Windows/OS start to load.
    • The VBR is a file with a specific file name in a specific location.
    • The VBR code is OS specific.
    • The VBR for Windows is a file in the root called: BOOTMGR
    • Volume boot record - Wikipedia
      • A volume boot record (VBR) (also known as a volume boot sector, a partition boot record or a partition boot sector) is a type of boot sector introduced by the IBM Personal Computer. 
      • The code in volume boot records is invoked either directly by the machine's firmware or indirectly by code in the master boot record or a boot manager. Code in the MBR and VBR is in essence loaded the same way.
  • Windows Boot Manager
    • Windows Boot Manager - Wikipedia
      • The Windows Boot Manager (BOOTMGR) is the bootloader provided by Microsoft for Windows NT versions starting with Windows Vista and Windows Server 2008. It is the first program launched by the BIOS or UEFI of the computer and is responsible for loading the rest of Windows. It replaced the NTLDR present in older versions of Windows.
      • The boot sector or UEFI loads the Windows Boot Manager (a file named BOOTMGR on either the system or the boot partition), accesses the Boot Configuration Data store and uses the information to load the operating system through winload.exe or winresume.exe on BIOS systems, and winload.efi and winresume.efi on UEFI systems.
      • On system with BIOS firmware, the BIOS invokes MBR boot code from a hard disk drive at startup. The MBR boot code and the VBR boot code are OS-specific. In Microsoft Windows, the MBR boot code tries to find an active partition (the MBR is only 512 bytes), then executes the VBR boot code of an active partition. The VBR boot code tries to find and execute the bootmgr file from an active partition.[3]
      • On systems with UEFI firmware, UEFI invokes bootmgfw.efi from an EFI system partition at startup, starting the Windows Boot Manager.
  • Operating System Boot Loader.

UEFI / MBR Boot Procedure

  • Windows MBR Boot Procedure
    • MBR is loaded into RAM by the firmware.
    • MBR searches for an active partition
    • MBR loads the Windows Boot Manager (BOOTMGR / VBR) from the active paritition.
    • Windows Boot Manager now loads the Windows Boot loader from one of these locations: 
      \windows\system32\winload.exe
\windows\system32\boot\winload.exe
    • Windows now loads
  • Windows EFI Boot Procedure
    • UEFI invokes the Windows Boot Manager from an EFI system partition (ESP) at startup, 
      \EFI\Microsoft\Boot\bootmgfw.efi
    • A user can make an OS selection now, or the default OS is loaded. These settings are defined in the BCD.
    • Windows Boot Manager now loads the Windows Boot loader from one of these locations. 
      \windows\system32\winload.efi
\windows\system32\boot\winload.efi
    • Windows now loads

Links

Image Formats

  • VHD (deprecated)
    • (Microsoft Virtual PC Virtual Hard Disk)
    • Azure (requires fixed size), Virtual PC, 
    • Developed by Microsoft 
    • Does Azure use VHD or VHDX? - Darwin's Data - Azure supports both VHD and VHDX disk formats for virtual machine disks. The choice between VHD and VHDX depends on the specific scenario and requirements.
    • VHD should not be used for new systems.
    • What is a VHD File? (Unpacking Virtual Hard Disk Mysteries) - Discover how Virtual Hard Disk (VHD) files revolutionize computing by simplifying management of multiple operating systems and enhancing efficiency.
  • VHDX
  • VMDK
    • (VMware Virtual Disk File)
    • VMware and ESXi
  • VDI
    • (VirtualBox Virtual Disk File)
    • VirtualBox
  • QCOW (deprecated)
    • KVM type Virtual Machines (old format, deprecated)
  • QCOW2
    • KVM type Virtual Machines
  • HDD,HDS
    • (Parallels Desktop Virtual Disk File)
    • Parallels
  • DD
    • These are created by the Linux command dd.
    • This is a binary blog, made by copying the RAW bits, and therefore is file system agnostic.
  • IMG
    • (Disk or Partition Image File)
    • This binary blog, made by copying the RAW bits, and therefore is file system agnostic.
    • IMG (file format) - Wikipedia
      • The .img file extension was originally used for floppy disk raw disk images only. A similar file extension, .ima, is also used to refer to floppy disk image files by some programs. A variant of raw image, called IMZ, consists of a gzipped version of a raw floppy disk image. These files use the .imz file extension, and are commonly found in compressed images of floppy disks created by WinImage.
      • QEMU uses the .img file extension for a raw image of a hard disk drive and calls the format "raw".
    • An .img file is just a raw disk image, a sector-by-sector copy of a storage device (like a hard drive, SSD, USB stick, etc.). It doesn’t have a special format; it’s literally just the binary contents of a disk.
    • .img was not a format invented by dd, it’s just a raw image file — a byte-for-byte copy of a disk.
  • IMA
    • (Virtual Floppy Disk Drive)
    • This is a binary blog, made by copying the RAW bits, and therefore is file system agnostic.
  • IMZ
    • GZipped (Virtual Floppy Disk Drive)
  • RAW
    • This is a binary blog, made by copying the RAW bits, and therefore is file system agnostic.
  • BIN
    • This is a binary blog, made by copying the RAW bits, and therefore is file system agnostic.
  • WIM
    • Microsoft Windows Image File
    • This is used to hold a Windows operating system and various other task.
    • This is not strictly a Disk Image file but can behave like one.
  • PHVD
    • Paragon
  • PMFX
    • DiskGenius 
  • E01
    • EnCase
  • AFF
    • (Advance Forensics Format)
  • EWF
    • (Expert Witness Compression Format)

Software

V2V and P2V

There will be cross over between software functionality and their category.

  • Conversion
  • Resizing
    • VhdResizer | Softpedia
      • A small and straightforward Windows utility designed to help users to shrink or expand the volumes of VHD files. Made in 2007.
      • How to Resize a Microsoft Virtual Hard Drive (VHD) File - When you create a Microsoft Virtual Hard Drive either through Virtual PC or Virtual Server, you have to specify the maximum size of the file up front.
    • VhdxTool
      • VhxdTool is a command line utility for Windows for quick creation and resizing of virtual hard disk files (VHDX).
      • This offers similar functionality to a former program vhdtool for Windows Server 2008/R2, but while vhdtool worked with older (VHD) format, vhdxtool works with the new (VHDX) format.
  • Compacting
    • vboxmanage
    • wslcompact
      • Compacts the size of the ever-growing WSL vhdx images.
      • Compacting the size of the ever-growing WSL images · microsoft/WSL · Discussion #9566 · GitHub
        • Give the usage commands of the tool.
        • Using virtual drives is a great idea. But, ext4 filesystem optimization inside VHDX images is not so well supported by the optimization tools. NTFS partitions are well optimized, but ext4 partitions are not so well optimized, and WSL suffers from this problem.
        • Since WSL doesn't provide a good solution for this, I have coded a small utility wslcompact that compacts the WSL partitions.
        • Any thoughts on improving the tool would be welcome. However, it would be better if the WSL team releases something similar. A good improvement would be an improvement in the export function, not only making a copy of the VHD file but dumping the contents in a newly created VHD, which would considerably reduce the size of the backup.
    • Optimize-VHD (Hyper-V Management Tools)
      • Needs Hyper-V to be installed. 
        Optimize-VHD -Path "Path\To\Your\VHDfile.vhdx" -Mode Full
    • DISKPART
      • Always present in Windows.
      • Optimize-Vhd doesn't require diskpart, of course, both processes can do more or less the same regarding drive compaction. Diskpart is somehow more convenient as it is readily available in all systems without the need for installing any extra package.
      • diskpart | Microsoft Learn - Reference article for the diskpart command interpreter, which helps you manage your computer's drives.
      • Compact a VHD file created by Disk Management? - Super User
        • You can then complete the compacting using diskpart from a Command Prompt
          1. Run diskpart
          2. Select the disk via its path: select vdisk file="<path>"
          3. Attach it as read-only: attach vdisk readonly
          4. Compact it: compact vdisk
          5. Upon completion of the compact, detach it again: detach vdisk
  • Image Creation (Disk)
    • DiskGenius: Data Recovery, Partition Manager, Backup & Disk Utilities
      • This software handles a variety of formats (.vmdk, .vhd, .vhdx, .vdi, .hdd, .img) but is read only in the free version, this means you cannot backup a drive to one of these formats with the free version. There is a proprietary DiskGenius disk image format (.pmfx) you can use for normal disk operations.
      • The paid version will also allow you to convert between the various Disk Image formats
      • You can also browse the files on disk images.
    • DiskImager
      • A simple but powerful tool for saving and writing disk images.
      • Save raw binary images of any disk as regular files.
      • Write disk images to physical disks.
      • When writing an image to a disk, supports industry-standard disk image formats (in addition to raw images), including VDI (VirtualBox), VMDK (VMware), E01 (EnCase), and VHDX/VHD (Virtual hard disk).
    • MultiDrive
      • 100% free disk management software for Windows. Clone, backup, restore, and erase drives with ease.
      • Allows you to create IMG of drives and I think also USB Drives.
    • Linux dd
      • This is a low-level command-line utility that copies data from one location to another, byte by byte. It's commonly used to:
        • Clone disks or partitions
        • Create disk images
        • Write ISO or IMG files to USB drives
      • How to Make Disk Images in Linux with the DD Command: An Essential Guide for Sysadmins – TheLinuxCode
        • Hey there! Do you manage Linux systems and need to make backups or clone disks? If so, the dd command should be part of your essential sysadmin toolkit. In this guide, I‘ll show you how to use dd to create complete disk images in Linux.
        • This is very comprehensive.
    • OSFClone
      • OSFClone is a free, open-source utility designed for use with OSForensics. OSFClone is a self-booting solution which lets you create or clone exact, forensic-grade raw disk images.
      • OSFClone is a free, self-booting solution which enables you to create or clone exact raw disk images quickly and independent of the installed operating system. In addition to raw disk images, OSFClone also supports imaging drives to the open Advance Forensics Format (AFF), AFF is an open and extensible format to store disk images and associated metadata, and Expert Witness Compression Format (EWF). An open standard enables investigators to quickly and efficiently use their preferred tools for drive analysis.
  • Image Creation (USB)
    • Rufus - The Official Website
      • Rufus is a small application that creates bootable USB drives, which can then be used to install or run Microsoft Windows, Linux or DOS.
      • This app also allows you to create disk image files (VHD/VHDX) of any USB pendrives plugged in.
        • These are fixed-size and raw, fully equivalent to a block-for-block disk image.
        • Rufus produces fixed (non-dynamic) VHD/VHDX images.
        • Fixed VHD/VHDX: Fully sized from the start, raw disk image with footer—this is what Rufus generates.
        • so reclaim the free space if you want to compact the drive, you woul dneed to convert to a Dynamix
      • Disk Format information
        • Why does Rufus call a .vhd file a DD Image? - Super User
          • Rufus developer here. This is because the save option is only for uncompressed VHD images, and uncompressed VHD images are just the same as DD images with an extra 512 byte footer added.
          • So, for all purposes, you can use DD to write the VHD image created by Rufus, and it will work just fine (with either the extra 512 byte footer being ignored as random data, if you use a larger sized disk, or being dropped, if using same exact same size disk).
          • Ergo, what Rufus creates when you click the save button is essentially a DD image since it can very much be used as such.
          • PS: For those who wonder why Rufus doesn't compress the VHD images it saves this is primarily due to the compression algorithm being used by Microsoft being very poor (IMO) and therefore not worth spending time implementing support for. This is also part of the reason why save as DD image is only available when clicking the advanced options.
        • Getting Rufus to output to a disk image file - Super User
          • Akeo
            • Rufus can create an uncompressed VHD image from whichever drive you have currently selected, which, for all intent and purposes can be used as a regular DD image (because the only difference with a regular DD image is an extra 512-byte footer, which, no matter how much you may think otherwise, will NEVER EVER come in the way of using the .vhd as a .img).
            • UPDATE: And recent versions can also create compressed VHDX images, that take less space, and that can work the same way.
          • gronostaj
            • Solution 1: Read the image from a prepared flash drive. Win32DiskImager can be used to do this.
            • Solution 2: Prepare your own hybrid ISO. Most of modern Linux distros use so called hybrid ISOs. They are valid ISOs that can be burned to a CD/DVD, but at the same time they are valid disk images that can be written to a flash drive. For example you could feed an Ubuntu ISO to Win32DiskImager and have it write it to disk (Rufus may offer this feature for some ISOs, it calls it "dd mode"). Community documentation on ISO customization is ten to thirteen years old, but may still be useful and you'll find many useful guides all over the Internet.
        • Make image from disc · Issue #1274 · pbatard/rufus · GitHub
          • Aha, then this is my cue to enlighten you: 
          • An uncompressed is just a DD image with a 512 byte footer added.
          • Therefore, because writing that footer (or not) is inconsequential, you can absolutely use the .vhd that was created by Rufus as a DD image.
          • Now, in case you have doubts about those extra 512 bytes potentially causing an issue let me explain that:
            • If you are restoring the image to a drive of the exact same size as the one you used to create the VHD, then these 512 bytes will happily be dropped by dd (At least the Linux version of dd doesn't care much if you try to write more data that the drive can hold - it'll report it, but it will not have any kind of impact on the data written), leaving you with a 1:1 copy of the original drive.
            • If you are using a drive that is larger than the one you used to create the VHD, then, even if you are unfortunate enough to have the 512-byte VHD footer fall at the precise location where the GPT backup is located (a few sectors before the end of the drive), then because this is a backup, any modern OS will detect that the backup is corrupted and fix it by duplicating the master, which is located at the beginning of the disk. Oh, and, even when you restore a pure GPT based DD image onto a larger drive, this kind of GPT backup fix needs to happen anyway, since, obviously, the backup GPT from the original image is at the wrong location, and a new backup GPT needs to be created at the right one. In other words, the writing of that 512-byte VHD footer is no different than the case of expecting whatever garbled data one might find at the expected location for the backup GPT when writing a "regular" DD image. And really, even as this is a complete non-issue, the possibility that the VHD footer will fall at the precise location where the backup GPT should go is exceedingly small. Oh, and if your image is using a MBR partition table, then it's even less of an issue, because none of the data up to (size of VHD - 512 bytes) will EVER be seen as accessible from the original image.
          • In summary, and I really have to stress this out, for all purposes, an uncompressed VHD image is exactly the same as a DD image. Therefore you can use the Rufus created .vhd exactly as you would use a standard DD image. There REALLY is no "What if...?" or "But surely..." associated to using an uncompressed .vhd as you would a DD image.
      • Disk image created by Rufus is shown as a growable image and not as a fixed disk
        • Why This Happens
          • Rufus does not create truly dynamic (growable) disks. Instead, it writes raw, fully allocated (fixed-size) disk images, even when saving to .vhdx format. But:
            • Rufus doesn't explicitly write the full VHDX metadata headers the same way Hyper-V or Windows Disk Management does.
            • This can cause third-party tools like StarWind to misinterpret the image format or default to "growable" in their interface or assumptions.
        • Evidence & Explanation
          • StarWind is designed to work with officially structured VHDX files (including all metadata).
          • Rufus writes uncompressed, flat VHDX files but doesn’t necessarily populate all optional VHDX metadata fields.
          • StarWind may detect the structure or size mismatch and label it as "growable" (which usually means "dynamic")—even if it's not actually dynamic.
        • How to Check What It Really Is
          • You can confirm the truth using:
          • PowerShell 
            Get-VHD -Path "C:\path\to\your.vhdx"
            • Look for VHDType: Fixed or Dynamic
          • vhdxinfo (from libvhdi) 
            vhdxinfo mydisk.vhdx
            • It will show you type: fixed or type: dynamic explicitly.
        • Workaround or Fix
          • If you want the VHDX to be 100% standards-compliant and recognized as "fixed" by all tools (including StarWind), do this:
            1. Mount the Rufus-generated VHDX in Windows.
            2. Create a new fixed-size VHDX using: 
              New-VHD -Path "C:\fixed-compliant.vhdx" -SizeBytes 10GB -Fixed
            3. Use dd or xcopy to clone contents to the new disk.
          • This gives you a 100% spec-compliant VHDX that should show as "fixed" in all tools.
    • ImageUSB
      • ImageUSB is an effective tool for writing an image to multiple USB Flash Drives for mass duplication and creating images of thoese drives.
      • ImageUSB is a free utility which lets you write an image concurrently to multiple USB Flash Drives. Capable of creating exact bit-level copies of USB Flash Drive (UFDs), ImageUSB is an extremely effective tool for the mass duplication of UFDs.ImageUSB also supports writing of an ISO file byte by byte directly to an USB drive (*). ImageUSB can also be used to install OSFClone to a USB Drive for use with PassMark OSForensics™.
      • Unlike other USB duplication tools, ImageUSB can preserve all unused and slack space during the cloning process, including the Master Boot Record (MBR). ImageUSB can perform flawless mass duplications of all UFD images, including bootable UFDs.
      • ImageUSB includes functionality to Zero a USB Flash Drive. This will replace the contents of the entire drive with 0s. Or alternatively to just Zero the MBR and/or GPT entries that exists on the drive. In addition, imageUSB has the ability to reformat even hard to format drives and reclaim any disk space that may be lost previously.
      • (*) CD ISO images use a different file systems compared to USB drives. So the direct imaging of ISO9660, Joliet or UDF file system, from a CD, to a USB drive, might not allow the USB drive to function in all operating systems. A reformat can recover the drive however. As of V1.5, imageUSB now supports extraction of ISO contents onto USB Drive.
      • Warning: Due to the forensic nature of image duplication by ImageUSB, please ensure that you select UFDs with a storage size similar to the image you wish to duplicate. For example, if a 2GB image is copied to an 8GB USB Flash Drive, the drive will only be able to use two out of the eight gigabytes of storage space. In this scenario, users will need to reformat the UFD in order to access the rest of the storage space.
    • USB Image Tool – alex's coding playground
      • USB Image Tool can create images of USB flash drives, SD cards and any other devices that are mounted as USB storage drives.
      • It can create an exact byte copy of these devices in raw and compressed image formats and also restores or write images on the device.
      • USB Image Tool works with any device, that implements the USB Mass Storage protocol. This includes flash drives, card readers/SD cards and a lot of other devices like digicams, cell phones and mobile music players.
    • USBImager
      • A really simple GUI application that writes compressed disk images to USB drives
        and creates backups. Available platforms: Windows, MacOSX and Linux. Its interface is as simple as it gets, totally bloat-free.

Disk Image Repair software

Other Software

  • sDelete (Secure Delete)
    • SDelete - Sysinternals | Microsoft Learn - Securely overwrite your sensitive files and cleanse your free space of previously deleted files using this DoD-compliant secure delete program.
    • How to use the command 'sdelete' (with examples) - Secure deletion is an essential part of data management and protection, particularly when it involves sensitive information. The command-line tool ‘sdelete’ offers a robust solution by permanently erasing files, directories, or entire volumes, thus ensuring that data cannot be recovered. Developed by Sysinternals, ‘sdelete’ employs advanced algorithms to overwrite data, mitigating the risk of unauthorized data access. Below we explore various use cases, each tailored to different secure deletion needs.
    • Meet Sdelete, the obscure Microsoft tool that wipes data for good | PCWorld - Sdelete, short for Secure Delete, is a powerful but little-known tool from Microsoft's Sysinternals suite that permanently wipes files and folders by overwriting data.
    • How to use SDelete to ensure deleted data is gone for good | TechTarget - Standard delete functions aren't good enough to guarantee data is gone for good. Learn how to use SDelete to ensure your information is removed permanently.

Mounting and Managing Images (in Windows)

  • Windows can mount VHD and VHDX natively.
  • Simple VHD Manager v1.5 - Simple Vhd Manager helps Virtual Hard Drive (VHD/VHDX) users to create and attach them or add them to boot menu.It is a Portable freeware
  • DAEMON Tools Lite: The most personal application for disc imaging yet - DAEMON-Tools.cc - DAEMON Tools Lite allows you to mount all known types of disc image files and emulates up to 4 DT + SCSI + HDD devices. It enables you to create images of your optical discs and access them via well-organized catalogue.
  • Arsenal Image Mounter - Arsenal Image Mounter mounts the contents of disk images as complete disks in Windows.
  • FTK Imager - Forensic Data Imaging and Preview Solution | Exterro
    • A free data preview and imaging tool used to acquire electronic evidence in a forensically sound manner by creating copies of computer data without making changes to the original evidence.
    • With FTK Imager, you can create forensic images of local hard drives, CDs and DVDs, thumb drives or other USB devices.
    • Mount an image for a read-only view that leverages Windows® File Explorer to see the content of the image exactly as the user saw it on the original drive.
  • OSFMount - Mount Disk Images & Create RAM Drives
    • The free OSFMount tool mounts raw disk image files in mulitple formats. You can also create RAM drives. Formats supported include img, dd, E01, VHD, ISO & bin
    • Instructions
      • Step 1 of 4: Just select your disk image, do not mount as ram disk
      • Step 2 of 4: Mount entire image as virtual disk
      • Step 3 of 4: Skipped by the software, ,ust be somethign to do with partitions
      • Step 4 of 4:
        • Read-only drive: Optional, but readonly keeps the original image clean
        • Write mode: Write cache
        • Drive emulation: Physical Disk Emulation = we need this for the boot sector
        • Drive type: HDD
      • NB:
        • In Starwind, your virtual disk will be listed as real disk called osfdisk
        • If you mount as FDD (possible a super floppy mode) Starwind will not see the drive.
    • OSFMount - What is the difference between logical and physical drive emulation modes:
      • In OSFMount, the difference between logical and physical drive emulation lies in how the mounted image is presented to the operating system:
        • Logical Drive Emulation
          • What it is: Mounts a partition within a disk image as a drive.
          • Use case: When you're only interested in accessing a specific partition (e.g., a file system like NTFS or FAT32).
          • Example: Mounting just the C: partition from a full disk image.
          • Visible as: A standard drive letter in Windows (e.g., D:, E:).
          • No boot sector or MBR is emulated—just the file system content.
          • Faster and simpler for common use cases like extracting files.
        • Physical Drive Emulation
          • What it is: Mounts the entire image as if it were a real physical disk, including the MBR, partition table, and all partitions.
          • Use case: When you need to work with the full disk structure (e.g., for forensic analysis or imaging software that requires MBR access).
          • Visible as: A physical disk (not a drive letter), usually accessed via disk management tools or forensic software.
          • Can emulate multiple partitions, raw sectors, boot records, etc.
          • Slower and more resource-intensive, but gives low-level access.
      •  Forum
        • OSFMount decides automatically whether to use physical or logical emulation based on the presence of a partition table in the image.
        • OSFMount shall automatically mount using Physical Drive or Logical emulation based on whether the image file contains a partition table or not
      • Another forum post clarifies the difference in practical terms:
        • Logical emulation: mounts only the file system as a volume — you get a drive letter but no low-level disk access..
        • Physical emulation: mounts the entire disk, including all sectors and partitions — useful for low-level forensic or VM boot tasks

DAEMON Tools Lite: The most personal application for disc imaging yet - DAEMON-Tools.cc

  • A well-known solution that allows you to mount, copy and create an image. It works with the most popular types of virtual discs. Download DT Lite to mount images without any limits!

Simple VHD Manager v1.5

  • Simple VHD Manager is portable freeware which helps VHD users simplify some of these operations:
    • You can attach and detach VHD/VHDX/ISO files via drag and drop
    • You can permanently attach a virtual hard disk in Windows 11 – 10 – 8,1 – 8 and Windows 7
    • You can easily add and/or remove VHD/VHDX/ISO files to the boot menu
  • Mount and Unmount a VHD File in Windows Explorer via a Right-Click
    • Virtualization has become an incredibly powerful and flexible way to deploy environments. So much in fact that Microsoft has integrated the ability to attach virtual hard drive (VHD) files as physical disks in the Windows Disk Management tool.
    • This process is easy enough to do manually but if you attach VHD files often then we have a solution which enables you to mount and unmount VHD files with a single click.
    • Thsi article has batch scripts you can attach to the context menu to allow easy use of the right click feature.

Best Practices

  • Fixed vs Dynamic disks - Microsoft Q&A - Hi Can anyone point me to a link that shows the correct official guidance from Microsoft as to if fixed or dynamic VHD's or VHDX's should be used in a production enviroment? Thanks

Tutorials

  • Create from Scratch
  • Export/Import Virtual Machines
  • Shrinking and Compacting
    • How to Shrink and Compact VHDX Files in Hyper-V? | Vinchin Backup
      • This article serves as a comprehensive tutorial on managing VHDX disk space in Microsoft Hyper-V environments.
      • It differentiates between shrinking and compacting VHDX files, outlines the advantages of the VHDX format over the older VHD, and provides detailed steps to perform both shrink and compact operations using both Hyper-V Manager and PowerShell commands.
    • How to Shrink and Compact Virtual Hard Disks in Hyper-V | TheWindowsClub - In this post, we will see how to shrink and compact hard disks in Hyper-V. This allows you to make room for the hard disks if they are too big for your physical computer. Since Hyper-V hard disks are usually dynamic in nature, it is crucial to shrink and compact virtual hard disks in Hyper-V.
    • How to Shrink the Virtual Disk of a Linux Distro in WSL | by bonguides.com | Medium - When you delete files or free up space in your WSL distribution, the virtual disk doesn’t automatically shrink. To reclaim this unused space, you need to compact the virtual disk (VHDX) file.
      • Diskpart 
        • Open PowerShell (Terminal) as administrator then type diskpart and press Enter
        • Use the following command, replacing the path with the actual location of your VHDX file to select the VHDX file. 
          select vdisk file=""
        • Once the disk is selected, run the below command within the DiskPart prompt to shirk the disk. This will reclaim the unused space in your VHD file. You should see a message indicating that the disk has been successfully compacted 
          compact vdisk
        • After the process is complete, you can check the size of the VHD file to confirm that it has been reduced.
      • Hyper-V
        1. Open PowerShell as administrator, then navigate to the location of the VHD file
        2. Run the following command to compact the VHD file: 
          Optimize-VHD -Path .ext4.vhdx full
        3. You can monitor the processing status on the screen. After completion, verify the virtual disk size to confirm functionality.
      • WSLCOMPACT
        • Install it using the below PowerShell commands: 
          Set-ExecutionPolicy RemoteSigned -Scope CurrentUser
iwr -useb https://raw.githubusercontent.com/okibcn/wslcompact/main/setup | iex
        • Run the wslcompact command to see the current size and estimated size of all WSL instances.
        • Run the below command to compact the disk and apply the change to the VHD file. 
          wslcompact -c ubuntu
        • If the tool is no longer needed, to remove the utility, close all your PowerShell instances, open a fresh one, and type: 
          Remove-Item "$($env:PSModulePath.split(';')[0])/WslCompact" -Recurse -Force
    • Shrink and Compact Virtual Hard Disks In Hyper-V (Guide) - If your Hyper-V environment starts running low on disk space, find out how to shrink and compact virtual hard disks in this step-by-step guide.
    • How to Shrink and Compact Virtual Hard Disks in Hyper-V - TechDirectArchive
      • Master the process of shrinking and compacting your Hyper-V virtual hard disks to save space and enhance virtual machine efficiency.
      • Different method iusing: Hyper-V Manager, PowerShell, Resize-VHD
    • WSL2 on Windows 11 - Virtual Disk vhdx is 200GB now | Reddit
      • Release VHDX unused space:
        Open Powershell and run the following:

wsl --shutdown

diskpart

select vdisk file="path-to-your-virtual-disk\ext4.vhdx"

attach vdisk readonly

compact vdisk

detach vdisk

exit
    • How to Compact/Shrink a WSL2 Virtual Disk – Stephen Rees-Carter
      • You can use the diskpart tool to compact a VHDX. This allows you to shrink a WSL2 virtual disk file, reclaiming disk space. I went from 100GB to 15GB.
    • How to Compact a WSL2 VHDX Virtual Disk | mycodde.blogspot.com - I was struggling to compact my WSL VHDX disk, it has hardly 10 GB data but it was showing 72 GB in size. 
  • Resizing
  • Starwind V2V Converter
  • VMware vCenter Converter Standalone
  • QEMU
  • VBoxManage
    • Converting disks (vboxmanage clonemedium)
      • Convert VMDK to VirtualBox VDI and compact disk - How to convert the vmdk to a VDI file and then compact it. 
        ## Convert
VBoxManage clonemedium disk --format VDI [drive]:[\path_to_file\sourceFile.vmdk] [drive]:[\path-to\destinationFile.vdi]

## Compact
VBoxManage modifyhd --compact "[drive]:\[path_to_image_file]\[name_of_image_file].vdi"
    • resize VDI image size 
      VBoxManage modifyhd "[drive]:\[path_to_image_file]\[name_of_image_file].vdi" --resize 81920
    • RAW Disk handling
      • vboxManage internalcommands createrawvmdk - Bing Search
        • The VBoxManage internalcommands createrawvmdk command in VirtualBox allows you to create a virtual disk that maps to a physical disk or partition. This is useful for accessing a physical disk directly from a virtual machine.
        • Example
          • To create a virtual disk that maps to an entire physical disk on a Windows host, use the following command: 
            VBoxManage internalcommands createrawvmdk -filename "C:\path\to\file.vmdk" -rawdisk \\.\PhysicalDrive0
          • This command creates a .vmdk file that represents the physical disk PhysicalDrive0
        • Accessing Specific Partitions
          • You can also map specific partitions of a physical disk. For example, to map partitions 1 and 5 of /dev/sda on a Linux host: 
            VBoxManage internalcommands createrawvmdk -filename /path/to/file.vmdk -rawdisk /dev/sda -partitions 1,5
          • This creates a .vmdk file that provides access to partitions 1 and 5 of /dev/sda
        • Important Considerations
          • Permissions: Ensure you have the necessary read/write permissions for the physical disk.
          • Data Safety: Incorrect use can lead to data loss. Avoid accessing the same partition from both the host and guest simultaneously.
          • Platform Differences: The device specification varies by platform (e.g., \\.\PhysicalDrive0 for Windows, /dev/sda for Linux).
        • Using VBoxManage internalcommands createrawvmdk, you can effectively utilize physical disks within your VirtualBox VMs, providing flexibility and direct access to hardware resources.
        • Learn more
      • Example Commands 
        vboxManage internalcommands createrawvmdk -filename "C:\Users\Public\Documents\VirtualBox\Raw Disk Images\spinrite.vmdk" -rawdisk "\\.\PhysicalDrive2"

vboxManage internalcommands createrawvmdk -filename "C:\rawdisks\spinrite.vmdk" -rawdisk "\\.\PhysicalDrive2"
      •  Use physical harddisk in Virtual Box - Straight forward instructions - Super User - Today, I managed to boot from physical drives. This achievement required....
  • How to Read VHDX Metadata (e.g., virtual size, block size)
    • PowerShell
      • Run: Get-VHD -Path "C:\path\to\your.vhdx"
        • This returns
          • VHD type (Fixed/Dynamic)
          • Size
          • Logical sector size
          • Physical sector size
      • Note: Only works on Windows with Hyper-V features installed.
    • Use vhdxinfo from libvhdi (Linux/macOS/Windows)
      1. Install libvhdi tools:
        • On Linux: sudo apt install libvhdi-utils
        • On macOS (via brew): brew install libvhdi
        • On Windows: Download libvhdi
      2. Run: vhdxinfo yourfile.vhdx
        • This will show:
          • Header info
          • Log GUIDs
          • Virtual disk size
          • Block size
          • Parent locator (if differencing disk)
    • Parse the VHDX File Manually (Hex or Script)
      • You can parse the structure using a hex editor or script:
        1. Look for the magic header at offset 0: 
          00000000  76 68 64 78 66 69 6c 65   # "vhdxfile"
        2. Read region table at 0x20000, which lists metadata regions.
        3. Use the region table to find:
          • Virtual disk size
          • Block size
          • Logical/physical sector size
          • GUIDs, versioning
      • You'd need to follow the Microsoft VHDX Specification to decode it fully.
  • TRIM'ing VHD(X) files
    • Optimize-VHD (Hyper-V) | Microsoft Learn
      • Optimizes the allocation of space used by virtual hard disk files, except for fixed virtual hard disks.
    • Optimize-Volume (Storage) | Microsoft Learn
      • Optimizes a volume.
      • The Optimize-Volume cmdlet optimizes a volume, performing defragmentation, trim, slab consolidation, and storage tier processing. If no parameter is specified, then the default operation will be performed per the drive type as follows:
        • HDD, Fixed VHD, Storage Space. -Analyze -Defrag.
        • Tiered Storage Space. -TierOptimize.
        • SSD with TRIM support. -Retrim.
        • Storage Space (Thinly provisioned), SAN Virtual Disk (Thinly provisioned), Dynamic VHD, Differencing VHD. -Analyze -SlabConsolidate -Retrim.
        • SSD without TRIM support, Removable FAT, Unknown. No operation.
    • Multiple VHD/VHDx Optimization using PowerShell Workflows
      • Optimize multiple VHD/VHDx files using PowerShell Workflows.
      • After about 5 minutes of investigation I came up with this simple PowerShell command:
        Get-VM | Where { $\_.State -eq 'Off' } | Get-VMHardDiskDrive | Optimize-VHD -Mode Full
      • It basically performs a full optimization on all VHD/VHDx files attached to all Virtual Machines that are in the Off state on the host the command is run on. This does the job quite well but has a few annoyances:
        • The optimization is performed in series.
        • Running guests won’t be optimized.
        • The command only works on VMs on the host the command is run on.

Convert and Compact an Offline VHD/VHDX Fixed Disk Image to a Dynamic Disk Image

  • TRIM is the process of the OS telling a SSD/NVMe drive to release the sector as it is no longer used, these drives do not actually zero fill the sectors to prevent uneeded writes, but in the future when the sector is read, the drive will supply back all Zeros.
  • In a Fixed Disk Image, an unused sector is denoted by being all Zeros.
  • In a Dynamic Disk Image, an empty unused sector is either all Zeros or is juts not referenced at all, in either case, all Zeros will be returned.
  • If the Disk Image files is part of an online system, the OS it is running can perform TRIM on the drives and volumes to "Zero out" unused sectors.

Instructions

VHD and VHDX are interchangeable for these commands.

  • Mount the Disk image 
    Mount-VHD -Path "D:\Path\To\Fixed-Disk.vhdx" -ReadOnly

Or

Mount through the GUI (Disk Management, Right Click)
    • -ReadOnly: This should protect the volume being written to by the OS, but not this command. I am not 100% on this.
  • Zero out free space (inside the VHDX) (TRIM)
    • You need to fill the free space with zeros so the compacting process knows it’s unused.
    • Use sdelete from Sysinternals: 
      sdelete64.exe -z X:
    • Replace X: with the drive letter assigned to the mounted VHD.
    • -z: This flag instructs ‘sdelete’ to write zeros over the available free space, a less intensive option than multiple passes and suitable for non-critical data.
    • This is needed because when a full RAW image of a drive is done, all bits are stored into the image, irrespective of whether that sector is holding any data.
  • Unmount the VHDX 
    • Dismount-VHD -Path "D:\Path\To\Fixed-Disk.vhdx"

Or

Unmount through the GUI (Disk Management, Right Click)
  • Convert the Fixed VHDX to Dynamic.
    • Powershell 
      Convert-VHD -Path "D:\Path\To\Fixed-Disk.vhd" -DestinationPath "D:\Path\To\Dynamic-Disk.vhd" -VHDType Dynamic
    • Use StarWind V2V / P2V Converter to convert the Fixed disk to a Dynamic disk
      • This process actually generates a copy of the disk image.
  • Compact the Dynamic VHDX.
    • Using Powershell 
      Optimize-VHD -Path "D:\Path\To\Dynamic-Disk.vhdx" -Mode Full
    • Using Diskpart 
      diskpart
select vdisk file="D:\Path\To\Dynamic-Disk.vhdx"
attach vdisk readonly
compact vdisk
detach vdisk
exit
      • readonly: This should protect the volume being written to by the OS, but not this command. I am not 100% on this.
  • Done

Running Trim in a live OS

  • Virtual Drive TRIM - VergeOS Docs - How to regain space on a virtual SSD
    • After importing a virtual machine from another hypervisor, sometimes the free space available inside the virtual machine does not match the free space reported to the VergeOS platform.
    • This discrepancy is often due to the virtual disk being thick-provisioned from the VM source, making VergeOS unaware of the unused disk space.
    • To resolve this, a TRIM/UNMAP operation needs to be performed on the virtual disk from within the virtual machine.
    • Trimming a Windows Drive 
      Optimize-Volume -DriveLetter YourDriveLetter -ReTrim -Verbose

Optimize-Volume -DriveLetter E -ReTrim -Verbose
    • Trimming a Linux Drive 
      ## Check TRIM Timer/Schedule Status:
sudo systemctl status fstrim.timer

## Check TRIM Service Status:
sudo systemctl status fstrim

If TRIM is enabled, an operation will run at the next scheduled time. If TRIM is not enabled, you can

## Run a manual TRIM using:
sudo fstrim -av

## Info
It is recommended to enable automatic TRIM to ensure that data usage is reflected accurately between VergeOS and the guest OS.

## To enable automatic TRIM, run:
sudo systemctl enable fstrim.timer

Repairing Disk Images

Partition Types

Makes your USB drive appear as a Hard drive and not removable Media

Is htis best ehre or antoher article

TL;DR = Might not be possible any more, and if it is, it is not worth the hassle.

  • Flip the Removable Media bit (RMB) on the drive itself
    • The Lexar BootIt utility was designed to "flip the removable media bit" on certain USB flash drives, making them appear as fixed (hard) disks to Windows. This was a niche utility and often highly specific to particular flash drive controllers. It's largely outdated and generally not recommended due to potential instability and the possibility of bricking your USB drive.
      The "removable media bit" is often a hardware-level setting within the USB drive's controller firmware. So, if a utility like Lexar BootIt doesn't work for your specific drive, there's often no generic software alternative that can reliably change this "bit."
    • Flip Your Bit USB Utility To Make Local Drive | getusb.info
      • BootIt is a Lexar based utility to flip the Removable Media Bit setting of a USB drive. What this means, is you can take a Lexar drive (and many other brands) and make it appear as a Local Drive on your PC rather than Removable Storage.
    • Removable USB Flash Drive as Local HDD in Windows | Windows OS Hub
      • In this article we’ll show you how to make a USB flash drive or an SD card to be identified in the Windows as a common local hard disk.
      • Probably, you may ask why it’s necessary? The matter is that by default Windows identifies all the USB flash drives and SD cards as removable drives that can’t be split into several partitions with the built-in Windows tools.
      • And even if you partition the USB flash drive into two or more volumes using third-party utilities (say, in Linux), only the first partition will be available in Windows. Windows supports multiple partitions only for hard disk drives identified as local (i.e. non-removable).
      • Windows recognizes USB flash drives as removable devices due to the presence of a special descriptor bit RMB (removable media bit) on each of the devices. If the system determines that RMB=1 when polling the connected device using StorageDeviceProperty function, it concludes that this device is a removable drive. Thus, in order to convert the USB-flash to the hard disk it is enough to modify this descriptor. 
      • Software covered
        • Lexar BootIt
        • Hitachi filter driver
    • How to make a disk drive "removable" - Windows 10 Forums - No meta description
  • Windows Registry
    • The mode cannot be changed in the Windows registry.

 

Published in Virtual Machines
Read more...
Friday, 18 July 2025 13:11

My DiskGenius Notes

My DiskGenius Notes

Booting DiskGenius from Ventoy

To boot DiskGenius from Ventoy, we need to generate the WinPE Bootable USB Drive and image it. Ventoy allows the use of several different image formats, all with their own strengths.

Make the DiskGenius Bootable WinPE USB

Create WinPE Bootable USB Drive - DiskGenius - DiskGenius supports to make bootable USB disk based on Windows PE so that you can recover lost data, manage disk or backup data when system cannot boot.

  • When you create your USB drive I think the build process uses your PC for the drivers and WinPE files so the resulting data size on the USB drive will change depending on your system.
  • I don't know if the DiskGenius USB drives are tried to the PC they were created on.

Format the USB drive

Although DiskGenius can use a USB drive in either the "Supper Floppy" (FDD) or Hard drive mode (HDD) it is easier to work with a HDD version where it has an MBR and Partition table like a traditional hard drive.

When you format a USB drive in Windows the "Super Floppy" format is used by default. I am not sure what happens when you use Windows to format a USB drive taht has a partition table, perhaps ot leaves it alone.

  • Format your USB in Rufus in MBR format

    • Boot selection: Non bootable
    • Partition scheme: MBR
    • Target system: BIOS or UEFI
    • Volume label: DISKGENIUS (or whatever you want)
    • File system: FAT32
    • Cluster size: Default (this might vary depending on the size of the USB drive)
  • Set the partition active
    • This is not required because DiskGenius does this when it makes the USB a WinPE drive.

VHD/VHDX (Recommended)

Method 1 - Disk Image of the USB

  • I would recommend getting a 2GB USB drive to keep the image size down and reduce the number of steps required.
    • (Win10 WinPE = 550mb)
    • (Win11 WinPe = 1GB+)
  • The size difference between the images created by these formats is minimal.
  • Aquire the Image
    • Connect the DiskGenius USB drive
    • Image with Rufus to get an Fixed Disk Image (VHDX) of the USB drive
      • Create a VHDX image, save it to your desktop and call it: FIXEDSIZE-DiskGenius v6.0.1.1645.vhdx
    • Disconnect the DiskGenius USB drive (optional)
  • Prepare the Image (optional, you can just use the image you have, as is)
    1. Mount the image (with native windows or OSMount)
    2. Run sdelete to zero out unused space on the USB drive's volume
    3. Use StarWind V2V / P2V Converter to generate a new dynamic (growable) vhdx image
      1. Call the new disk image: DiskGenius v6.0.1.1645.vhdx
    4. Compact the Dynamic disk image to reduce space
  • Configure Ventoy
    • Connect the Ventoy USB drive
    • Copy DiskGenius v6.0.1.1645.vhdx to your ventoy images partition.
    • Install the Ventoy Windows VHD Boot Plugin
      • Plugin.WinVhdBoot . Ventoy
        • Ventoy uses this plugin to boot VHD(x) files with Windows 7+ in it.
        • Both Legacy BIOS and UEFI are supported. Both fixed and dynamic VHD(x) are supported.
        • In UEFI mode, only 64 bit Windows is supported, 32 bit is NOT supported.
  • Done
    • Boot from your Ventoy USB and check it loads.

Method 2 -  Manually Create VHDX

  • Create the VHDX (there are probably other ways)
    • Open up `Disk Management`
      • Run: diskmgmt.msc
    • Action --> Create VHD
      • Use these settings:
        • Location: C:\Users\quantunwarp\Desktop\DiskGenius v6.0.1.1645.vhdxDiskGenius v6.0.1.1645.vhdx
        • Virtual hard disk size: 2GB
        • Virtual hard disk format: VHDX
        • Virtual hard disk type: Dynamically expanding (Recommended)
    • Once created, the drive is automatically mounted but needs to be intialised like any new disk.
      • In `Disk Mangement` Right click on the new drive and select Initialise Disk
      • Use the MBR partition style
    • Create a FAT32 partition
      • Right click on the drive and select New Simple Volume and follow the wizard.
      • Use these settings:
        • File System: FAT32
        • Allocation unit size: Default
        • Volume Label: Disk Genius
        • Perform a quick format.
    • Set the partition active
      • Use DiskGenius to set the partition active
  • Prepare the Image
    • Mount the VHDX disk image.
    • Connect the DiskGenius USB drive.
    • Show hidden files
    • Copy all files on the USB drive to the VHDX volume.
    • Hide all files again
    • Unmount/Detach the Disk Image.
  • Configure Ventoy
    • Connect the Ventoy USB drive
    • Copy DiskGenius v6.0.1.1645.vhdx to your ventoy images partition.
    • Install the Ventoy Windows VHD Boot Plugin
      • Plugin.WinVhdBoot . Ventoy
        • Ventoy uses this plugin to boot VHD(x) files with Windows 7+ in it.
        • Both Legacy BIOS and UEFI are supported. Both fixed and dynamic VHD(x) are supported.
        • In UEFI mode, only 64 bit Windows is supported, 32 bit is NOT supported.
  • Done
    • Boot from your Ventoy USB and check it loads.

WIM

  • This only works for UEFI/EFI environments.
  • If you try and boot the WIM when using MBR/CSM/Legacy it will start to load and then a black screen will overlay the Ventoy loader and it will crash.
  • Aquire the Image
    • Connect the DiskGenius USB drive.
    • Show hidden files.
    • Copy the following file to your desktop: ...\DiskGenius - Boot\boot.wim
    • Disconnect the DiskGenius USB drive (optional)
  • Prepare the Image
    • Rename boot.wim to something human readable like DiskGenius v6.0.1.1645.wim
  • Configure Ventoy
    • Connect the Ventoy USB drive
    • Copy DiskGenius v6.0.1.1645.vhdx to your ventoy images partition.
    • Install the Ventoy Wimboot Plugin
  • Done
    • Boot from your Ventoy USB and check it loads.

ISO

  • This is a very trick and time consuming option and is not really needed
  • Try and find and ISO already done, and then use this.

method 1 get a windows disk and swap out the contents, keeping the boot sector

  • Aquire the Image
    • Connect the DiskGenius USB drive
    • ..............
  • Prepare the Image
    • ...........
  • Configure Ventoy
    • Connect the Ventoy USB drive
    • Copy DiskGenius v6.0.1.1645.iso to your ventoy images partition.
  • Done
    • Boot from your Ventoy USB and check it loads.

 

Starwind conversion notes (to reduce size)

  • Mount your drive image
    1. Windows Native (GUI)
      1. Right click on the drive image and select mount.
      2. or through `Disk Management`: Action --> Attach VHD
    2. Windows Native (diskpart)
      1. Select the disk via its path: select vdisk file="<PathToYourDesktop>\DiskGenius v6.0.1.1645.vhdx"
    3. OSFMount
  • Convert the image with starwind
    • Select the location of the image to convert: P2V
    • Select physical data to convert: Physical Disk
    • Select physical disk to convert: Virtual Disk
    • Select the location of the destination image: Local file
    • Select the destination image format: VHD/VHDX
    • Select option for VHD/VHDX image format: VHDX growable image
      • Ignore `Activate windows Repair Mode`
    • Set destination filename: <PathToYourDesktop>\DiskGenius v6.0.1.1645.vhdx
    • Click `Convert`
  • Unmount your drive imagwe
    • Window Native (GUI)
      • You can unmount by right clicking on the drive and selecting Eject.
      • or through `Disk Management`: Right click on the drive you want to unmount and select, Detach VHD.
    • Windows Native (diskpart)
      • run the command: detach vdisk

When you Mount a DiskGenius USB Disk Image as a hard drive in Windows you get this error

  • TL;DR this error is caused by the source USB formatted as a "Super Floppy"
  • Starwind P2V does not support "Super Floppy" for live copying as it expects a partition table.
  • A USB drive can be formatted as a traditional hard drive (HDD) or in a "Super Floppy" format akin to a traditional floppy disk (FDD).
  • "Super Floppy" does not have a MBR or a Partition Table which is what Starwind is looking for when copying from the mounted Disk image.
  • Starwind wiil not present/see Removable Media disks or their Volumes only hard disks.

Scenario

  • You used a large USB drive for your DiskGenius WinPE and as a consequence the image is very large even though most of the space is empty.
  • You mounted the drive image using either Windows Native or OSFMount. (in this example we has used OSFMount)
  • The drive has successfully mounted
  • You open Starwind V2V to convert your large USB Fixed size disk image to a growable VHDX image which will create a much smaller image file
  • After going through the wizard, you click on convert and almost immediately you get the following error.

Error

 

Log File

7/21 16:24:49.493 2b68 Facility ???INFO: CV2V_ConverterApp::InitInstance: SetDllDirectory: C:\Program Files\StarWind V2V Converter\lib
7/21 16:24:49.493 2b68 Facility ???INFO: CV2V_ConverterApp::InitInstance: File Version: 9.0.1.757
7/21 16:24:49.508 2b68 Facility ???INFO: CV2V_ConverterApp::InitInstance: Load language IDR_XML_ENGLISH
7/21 16:24:49.508 2b68 Facility ???INFO: CV2V_ConverterApp::InitInstance: Set current language English
7/21 16:24:49.508 2b68 Facility ???INFO: CPS_V2V_Converter::CPS_V2V_Converter: Version MajorVersion 10, MinorVersion 0, BuildNumber 19045
7/21 16:24:49.508 2b68 Facility ???INFO: WrapperHyperV::WrapperHyperV: LoadLibrary C:\Windows\system32\wbem\wmiutils.dll - 94cb0000
7/21 16:24:49.508 2b68 Facility ???INFO: WrapperHyperV::WrapperHyperV: LoadLibrary C:\Windows\system32\wbem\wmiutils.dll - 94cb0000
7/21 16:24:53.723 2b68 Facility ???INFO: ICPPage::OnWizardNext: SetActivePage - eSynchronization
7/21 16:24:57.926 2b68 Facility ???INFO: ICPPage::OnWizardNext: SetActivePage - eAzureConnectionTo
7/21 16:25:06.162 2b68 Facility ???INFO: ICPPage::OnWizardNext: SetActivePage - eLocationTo
7/21 16:25:09.053 2b68 Facility ???INFO: ICPPage::OnWizardNext: SetActivePage - eFormat
7/21 16:25:12.381 2b68 Facility ???INFO: ICPPage::OnWizardNext: SetActivePage - eOptionsHyperV
7/21 16:25:17.914 2b68 Facility ???INFO: ICPPage::OnWizardNext: SetActivePage - eConverting
7/21 16:25:43.043 2b68 Facility ???INFO: CPPOutputFile::onNext: Out path: C:\Users\quantumwarp\Desktop\DiskGenius-Dynamic.vhdx, Out format 11
7/21 16:25:43.043 2b68 Facility ???INFO: ICPPage::OnWizardNext: SetActivePage - eHYPERVConnectionTo
7/21 16:25:43.043 1788 Facility ???INFO: CPPConverting::Convert: InConvertType: 2, OutConvertType: 4
7/21 16:25:43.043 1788 Facility ???INFO: CPPConverting::ConvertDisks: pathInput: 1, pathOutput: C:\Users\quantumwarp\Desktop\DiskGenius-Dynamic.vhdx
7/21 16:25:43.043 1788 Facility ???INFO: P2V::CreateInputImage: path 1
7/21 16:25:43.043 1788 Facility ???INFO: P2V::CreateInputImage: Size 1 - 15518924800, BytesPerSector - 512
7/21 16:25:48.090 1788 Facility ???INFO: CPPConverting::ConvertDisks: CreateInputImage: 0
7/21 16:25:48.090 1788 Facility ???INFO: CPPConverting::ConvertDisks: CreateOutputImage: 0
7/21 16:25:48.090 1788 Facility ???ERROR: CPPConverting::ConvertImage: Read 3
7/21 16:32:00.054 1788 Facility ???ERROR: CPPConverting::ConvertDisks: ConvertImage - 2
7/21 16:32:00.054 1788 Facility ???ERROR: CPPConverting::Convert: FAILED

Log Explained

Here's what the log tells us:

  • StarWind V2V Converter Version: 9.0.1.757 - This is a relatively recent version, so it's unlikely to be a bug in an old release.
  • Conversion Type:
    • InConvertType: 2 - This indicates a Physical Disk source.
    • OutConvertType: 4 - This indicates a Hyper-V (VHDX) destination.
    • This confirms that StarWind V2V Converter sees the OSFMounted IMG as a physical disk, which is good.
  • Source Disk Identification: pathInput: 1 and Size 1 - 15518924800, BytesPerSector - 512
    • pathInput: 1 likely refers to the "Disk 1" as seen by Windows Disk Management. This corresponds to the virtual disk presented by OSFMount.
    • The size 15,518,924,800 bytes (approx 14.45 GB) and BytesPerSector - 512 are correctly identified by the converter from the virtual disk.
  • Failure Point:
    • CPPConverting::ConvertDisks: CreateInputImage: 0 - This line means StarWind successfully initialized the connection to the source disk.
    • CPPConverting::ConvertDisks: CreateOutputImage: 0 - This line means StarWind successfully initialized the connection to the destination VHDX file.
    • CPPConverting::ConvertImage: Read 3 - This is the critical error. It happens immediately after the input and output images are created, right at the point where StarWind tries to read the actual data from the source disk. This confirms a low-level read error from the OSFMounted virtual disk.
    • The subsequent errors (ConvertImage - 2, FAILED) are a direct consequence of this initial read failure.
  • Conclusion from the log
    • The log strongly indicates that the issue is not with StarWind V2V Converter itself, nor with the destination drive or VHDX creation. The problem lies squarely with StarWind's ability to read data from the virtual disk presented by OSFMount, which is derived from your disk image file created from your DiskGenius USB drive.
    • Rufus can write raw ISO or DD images that may not have a typical partition table or disk signature that StarWind expects. This can cause problems when it tries to interpret sectors and fails.
  • Additional
    • This following error (not in the example above) shows an issue with the drive (VHD or physical) not being able to read sectors. If it is a physical drive then this indicates the drive is failing. 
      10/21 12:55:57.929 2764 Facility ???WARNING: P2V::getBuffer: ReadFile - 23. Data error (cyclic redundancy check).
10/21 12:55:57.929 2764 Facility ???ERROR: CPPConverting::ConvertImage: Read 3
10/21 13:03:16.650 2764 Facility ???ERROR: CPPConverting::Convert: FAILED

 

Background

Cause

After in-depth research I can clarify the error is not cause by:

  • Rufus or DiskImager: these apps just create a binary image of the disk and change nothing. Any errors or issues of the drive are blindly transferred to the disk image.
  • Windows 10 or Windows 11: Apart from the size of the boot.wim there is no difference (Win10 = 550mb)(Win11 = 1GB).
  • Windows Native and OSFormat disk image mounting: These were mounting the disk image correctly and giving a 1:1 representation of the imaged USB drive.
  • The disk image being mounted as "Read-Only": This does not interfer with the conversion as it is used as a source only.
  • The size/capacity of the USB drive: None of the software has any size limits, certainly none that have been reached.
  • Cluster Size / Allocation Unit Size: I tried both 4096 and 8192 and those made no difference. 8192 seems to be the default allocation size for 16GB USB drives.
  • Trying different locations for the source and destination files made no difference.
  • VHD, VHDX and IMG disk image formats all worked as expected.
  • I got no Windows Defender errors, so in this case it was not the AV causing issues.
  • Encryption on the drive can cause issues but these drive had none so can be ruled out.

What I have tried

  • Zero filling the USB before reformatting = When I formatted with Windows, it just created a "Super Floppy" USB drive because this is how Windows formats them.
  • I check the USB drive for errors by doing a full surface scan, it found no errors.
  • I checked the FAT32 file system with chkdsk which found no errors.

Diagnostics

How to check if your USB is configured as "Super Floppy" or a traditional Hard drive (HDD)

There are a couple of methods outlined below.

  • diskpart
    • Run the following commands (swap # for the number assigned to your USB disk): 
      diskpart
list disk
select disk #
detail disk
    • This should give an output as follows: 
      Microsoft DiskPart version 10.0.26100.1150

Copyright (C) Microsoft Corporation.
On computer: WIN11PC

DISKPART> list disk

  Disk ###  Status         Size     Free     Dyn  Gpt
  --------  -------------  -------  -------  ---  ---
  Disk 0    Online         1863 GB  4096 KB        *
  Disk 1    Online         7651 MB      0 B

DISKPART> select disk 1

Disk 1 is now the selected disk.

DISKPART> detail disk

General USB Flash Disk USB Device
Disk ID: 6F20736B
Type   : USB
Status : Online
Path   : 0
Target : 0
LUN ID : 0
Location Path : UNAVAILABLE
Current Read-only State : No
Read-only  : No
Boot Disk  : No
Pagefile Disk  : No
Hibernation File Disk  : No
Crashdump Disk  : No
Clustered Disk  : No

  Volume ###  Ltr  Label        Fs     Type        Size     Status     Info
  ----------  ---  -----------  -----  ----------  -------  ---------  --------
  Volume 5     E   PICTURES     FAT32  Removable   7651 MB  Healthy

DISKPART> list partition

  Partition ###  Type              Size     Offset
  -------------  ----------------  -------  -------
* Partition 1    Primary             14 GB      0 B

DISKPART> select partition 1

There is no partition selected.

DISKPART>
    • As you can see, Disk 1 (the USB):
      • It is not a GPT based disk.
      • Has no free space, meaning that all the space has been assigned to partitions.
      • It is not a "Boot Disk".
      • Has only 1 Volume/Partition, which is not bootable.
      • When you try and select the partition, diskpart returns the error, There is no partition selected., indicating it has an error accessing the partition table, which do not exist on "Super Floppy" USB drives.
    • This is a "Super Floppy" USB drive, or the partition table is corrupted.
  • DiskGenius
    • This images shows you a 16GB "Super Floppy" USB drive

      1. Basic/RAW = This means there is no MBR or Partition Table.
      2. FAT32 (FDD) = This is a FAT32 partition, but has not been defined in a partition table.
      3. MBR Signature: 00000000 = No MBR is defined.
      4. Partition Table Style: NONE = No partition table.
      5. Volume/Partition is not marked as active.
      6. In the drive tree, there is a floppy icon instead of a HDD.
  • OSFMount
    • Does not show partitions in "Step 2 of 4" as shown below, and "Mount partitions as virtual disks" is greyed out.

      • This indicates a "Super Floppy" or corrupt disk image.
      • There is no valid partition table for OSFMount to read.
      • If the Volume gets mounted in windows then the Image is not corrupt and the drive image is that of a "Super Floppy".
    • I have an image that when i mount as a physical disk shows not partitions in OSFMount but windows mounts a volume form this drive?
      • This behavior can be confusing, but it usually comes down to how Windows interprets disk images versus how OSFMount handles physical vs logical mounting.
      • Your Situation:
        • You mount an image in OSFMount as a physical disk.
        • OSFMount shows no partitions in its interface.
        • Windows still assigns a drive letter and allows you to access a volume from that disk.
      • Likely Explanation:
        • Even though OSFMount is mounting the image as a physical drive, it may not parse or display the partitions correctly, especially if:
          • The partition table is non-standard or corrupt.
          • The image is just a raw partition, not a full disk with MBR/GPT.
          • OSFMount has trouble recognizing certain filesystems or partition layouts.
          • The image has a "superfloppy" format — a single partition image without an MBR (common in some forensic images or disk clones).
        • Yet Windows is often more lenient:
          • It scans all volumes and can mount a recognizable file system (e.g., NTFS or FAT32), even if the partition table is missing or broken.
          • Windows may treat the image like a full partition, even though OSFMount expects a structured physical disk.
      • What You Can Do:
        • Try mounting the image as a logical drive instead of physical:
          • In OSFMount, choose: Mount as drive letter (logical drive)
          • Select the image file and choose offset 0 (or auto-detect).
          • This usually works well if the image is a raw partition.
        • Use disk utilities to inspect the structure:
          • Tools like FTK Imager, WinHex, or HxD can help determine if the image has an MBR, GPT, or just a raw file system.
          • You can also check in Disk Management (diskmgmt.msc) how Windows sees the mounted image (does it show a healthy volume or an unallocated disk?).
        • Check the image format:
          • If it’s from forensic software (like EnCase or dd), it might be a raw partition or use non-standard formatting.
      • Quick Test
        • If you suspect it's a partition image (not a full disk), here’s a trick:
          • Open it in OSFMount.
          • Mount it as a logical drive with Offset = 0 and file system type = "Auto-detect".
          • If it mounts and shows files, it’s just a partition, not a full disk with an MBR.

Workarounds

Solutions

  • Always use Dynamic/Growable disk image files where possible to keep space used down.
  • Always use MBR/HDD style formatting on your USB drives where possible.
  • Use a HDD/MBR Style USB drive to begin with or make one:
    • Using Rufus, convert the USB drive into bootable USB, with a single FAT32 partition (HDD format), and of course the partiontion will be marked as `Active`.
    • Then use DiskGenius to make it a WinPE.
    • This way ensures the USB drive and any images created from it will work in Starwind P2V as there is a partition table present.
    • This USB drive will is NOT a "Super Floppy".
  • Use the original disk image as is.
    • You can keep the image size down by using a smaller USB drive (Win10 = 1GB, Win11 = 2GB).
  • Use DiskGenius Pro to make a direct VHD/VHDX image from the USB drive
    • This is the same as using Rufus to create the disk image.
    • All disk images created by DiskGenius are of the Fixed size type.
  • Build your own VDH/VDHX image
    • Manually create a VHD/VHDX image on your PC (MBR, Single 2GB FAT32 partition)
    • Mount the VHD/VHDX drive image
    • Set the partition as `Active`.
    • Then manually copy all of the files from the USB drive into the new partition ion the drive image.

 

Other DiskGenius Notes

  • Docs
  • Misc
    • When you mount a drive in Windows either using the GUI or diskpart, the image will be mounted as a physical drive.
    • Windows and OSFMouint will mount an image whether is is a normal HDD style or a "Super Floppy" (FDD) style image.
    • Windows will not fall back to a logical volume emulation if a VHD (Virtual Hard Disk) image file is corrupted—even slightly.
    • can 7zip open VHD files?
    • The partition table is not inherently protected—it's just stored in a specific location on the USB drive. While it can’t be easily changed by average users without intent or privilege, it can be accessed, overwritten, or corrupted by software that has sufficient access.
    • All VHDs (and other image formats) created with DiskGenius, are fixed-size containers from the outset, made via sector-level cloning.
    • The `Convert Boot Mode` options have moved here:
      • Disk --> Convert Boot Mode -->
        • Convert To USB-ZIP Bootable Mode
        • Convert To USB-FDD Bootable Mode
        • Convert To HDD mode
    • If a Volume's `File System` type is display in Red
      • This means it is set as the drive's active partition.
      • It should also be accompanied by an `Attribute` of A.
    • When you format a USB disk, windows formats it in "Super "loppy format. These can can be booted by a legacy BIOS, but these are not a bootable hard drive.
    • Making a USB drive "bootable" adds in an MBR which includes a partition table.
    • Windows when loading a USB checks for both a partition table and then if this is not present searches for a partition (Super Floppy) allowing for both types of USB to load in Windows.

 

WinPE - Low Resolution

  • TL;DR = UEFI GOP (Graphics Output Protocol) sets the screen resolution before WinPE when using EFI on my lenovo (resolution seems to be set at 800x600 and cannot be changed)
  • Solution = Boot in legacy mode and then change the resolution in DiskGenius if needed: View --> Set screen resolution (Options: 1920x1080, 1024x768). This was on my Lenovo laptop.
  • Notes
    • Using a different boot loader like Grub which can be used to see what resolutions the GOP provides.
    • My Lenovo on EFI returns a resolution of 800x600 but when I use CSM/Legacy I get 1024x768 and 1920x1080 which I can swap between with the DiskGenius menu in WinPE.
    • For most users on modern UEFI systems — the hard limit is 1024x768.
    • Some people say showing the boot logo helps increase screen resolution (not tested).

My Notes

When trying to change the resolution of my screen when using DiskGenius on my Lenonvo latop.

  • Legacy Boot = allows me to change res in WinPE
  • setres = no
  • qres = 2005, didnt work
  • powershell = no powershell to use
  • unattended.xml = doesn't work
    • Modify the WinPE Image Before Boot (Recommended):
      • This is the most reliable method for setting the desired resolution. You do this by customizing the boot.wim file that makes up your WinPE environment.
      • You can create an unattend.xml file with display settings and place it in the root of your WinPE ISO or USB. This file will be processed during the WinPE startup phase.
      • Here's an example of the unattend.xml snippet for setting resolution: 
        <?xml version="1.0" encoding="utf-8"?>
<unattend xmlns="urn:schemas-microsoft-com:unattend">
    <settings pass="windowsPE">
        <component name="Microsoft-Windows-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <Display>
                <ColorDepth>32</ColorDepth>
                <HorizontalResolution>1920</HorizontalResolution>
                <RefreshRate>60</RefreshRate>
                <VerticalResolution>1080</VerticalResolution>
            </Display>
        </component>
    </settings>
</unattend>
  • bcdedit / graphicsresolution
    • I made sure I was working on the WinPE drive: X: eg:
      X:\bcdedit /store D:\boot\BCD /set {globalsettings} graphicsresolution 1280x1024
    • Works 
      bcdedit /store "d:\boot\BCD" /set {bootmgr} graphicsresolution "800x600"
bcdedit /store "d:\boot\BCD" /set {bootmgr} graphicsresolution "1024x768"
    • Not Working 
      bcdedit /store "d:\boot\BCD" /set {bootmgr} graphicsresolution "1280x768"
bcdedit /store "d:\boot\BCD" /set {bootmgr} graphicsresolution "1280x1024"
    • Reason
      • Most UEFI firmware initializes the screen at 1024x768 or lower.
      • graphicsresolution is backed by the boot graphics adapter using VESA BIOS Extensions (VBE) or UEFI GOP (Graphics Output Protocol).
      • Changing that during early boot (before Windows loads its display drivers) is not officially supported past that limit.
      • For most users on modern UEFI systems — the hard limit is 1024x768.

 

 

Links

  • Standards
    • GOP - OSDev Wiki
      • This is the new standard for UEFI that superseded VESA (BIOS) and UGA (EFI 1.0).
      • It has basically the same functions as VESA, you can query the modes, set the modes.
      • It also provides an efficient BitBlitter function, which you can't use from your OS unfortunately.
      • GOP is an EFI Boot Time Service, meaning you can't access it after you call ExitBootServices(). However, the framebuffer provided by GOP persists, so you can continue to use it for graphics output in your OS.
  • Other Testing
    • WinPE Screen Resolution and DPI scaling on UEFI firmware | mistyprojects.co.uk
      • This article is based on observations on screen resolution and DPI scaling in WinPE.
      • Screen resolution can be changed from a running WinPE booted on BIOS firmware.
      • Changing the screen resolution on UEFI firmware is not so simple. Settings in unnatend.xml appear to be ignored and the screen resolution is fixed. Based on available information screen resolution appears to be set based on Graphics Output Protocal (GOP) mode. Once WinPE is booted, the resolution cannot be changed.
      • Setting the resolution in the BCD 
        bcdedit.exe /set {globalsettings} graphicsresolution 800x600
bcdedit.exe /set {globalsettings} graphicsresolution 1024x600
bcdedit.exe /set {globalsettings} graphicsresolution 1024x768
      • Attempting to use an unsupported resolution (unsupported by bcdedit.exe - not the hardware) resulted in the following output: 
        C:\windows\system32>bcdedit.exe /set {globalsettings} graphicsresolution 1920x1080
The integer data is not valid as specified.
Run "bcdedit /?" for command line assistance.
The parameter is incorrect.
      • The highestmode option is also supported in bcdedit.exe. This option will enable boot applications to use the highest graphical mode detected by the firmware 
        bcdedit.exe /set {globalsettings} highestmode on
      • To find supported resolutions, use the graphicsmode --info command on the GRUB4UEFI commandline.
  • unattend.xml
  • Ventoy
  • Software to change Resolution

 

 

 

 

 

 

Published in Applications
Read more...
Thursday, 17 July 2025 13:16

My Crashed MariaDB Server Notes

This is just a temporary article for these notes, I might move them.

Extract all failed tables from the error log with Notepad++

  • Get Setup 
    • Download the error log from: 
      /var/lib/mysql/server.quantumwarp.com.err
    • Open the log file in Notepad++
  • Remove all lines without Table errors
    • Open the Replace dialogue (Ctrl + H)
    • Configure as follows:
      • Find what: 
        ^(?!.*\[ERROR\] mariadbd: Table).*\r?\n
      • Replace with: nothing
      • Search Mode:
        • Regular expression
        • without `. matches newline`
      • Click Replace All
  • Make the lines Uniform
    • Start of the line
      • Open the Replace dialogue (Ctrl + H)
      • Configure as follows:
        • Find what: 
          .*\[ERROR\] mariadbd: Table '\.
        • Replace with: nothing
        • Search Mode:
          • Regular expression
          • without `. matches newline`
      • Click Replace All
    • End of the line (optional)
      • Open the Replace dialogue (Ctrl + H)
      • Configure as follows:
        • Find what: 
          ' is marked as crashed and should be repaired$
        • Replace with: nothing
        • Search Mode:
          • Regular expression
          • without `. matches newline`
      • Click Replace All
  • Sort the lines
    • Edit --> Line Operations --> Remove Duplicate Lines
    • Edit --> Line Operations --> Sort Lines Lexicographically Ascending
Published in MySQL
Read more...
Saturday, 31 May 2025 14:52

My Notepad++ Notes

 

Throughout these examples I have used the regex YourCodeHere to and you should change this to match your needs

Line Operations

Using a capture Group

Find:   ^.*YourCodeHere(.*)
Replace:  YourCodeHere\1

Add to the end of a line

  • Open the Replace dialogue (Ctrl + H)
  • Configure as follows:
    • Find what: $
    • Replace with: your content
    • Search Mode:
      • Regular expression
      • without `. matches newline`
  • Click Replace All

Notes

Multiline Search and Replace

Delete Specific lines

A couple of methods to identify lines and delete them.

Regex

  • Open the Replace dialogue (Ctrl + H)
  • Configure as follows:
    • Find what: 
      .*YourCodeHere.*\n
.*YourCodeHere.*\r?\n
    • Replace with: nothing
    • Search Mode:
      • Regular expression
      • without `. matches newline`
  • Click Replace All

Notes

  • The end of line character should be \n if straight from your Linux box, but you might just want to check this as it could be \r\n
  • YourCodeHere should be swapped for something on each of the lines you want to delete.
  • Notepad++ Delete Lines with Find & Replace - Super User
    • some useful end of line regex that might still be valid.

Bookmark Line Method

Delete all lines that don't contain ....

A couple of methods to identify lines and delete all the others.

Inverse Regex Replace

  • Open the Replace dialogue (Ctrl + H)
  • Configure as follows:
    • Find what: 
      ^(?!.*YourCodeHere).*\r?\n
^(?!.*YourCodeHere).*\R

# When this is used, blank lines are left
^((?!YourCodeHere).)*$
    • Replace with: nothing
    • Search Mode:
      • Regular expression
      • without `. matches newline`
  • Click Replace All

Bookmark Line Method

  • Select the Mark tab from the replace dialogue (Ctrl + H)
  • Enter your Regex to match code on the lines you want to keep. 
    YourCodeHere

# This will allow marking of all lines not containing [ERROR]
^((?!YourCodeHere).)*$
    • Your Regex does not need to match all of the content on the line, just something unique to allow identification.
  • Check Bookmark line.
  • Search Mode:
    • Regular expression
    • without `. matches newline`
  • Click Mark All.
  • Menu --> Search --> Bookmark --> Remove Non-Bookmarked Lines
  • In the replace dialogue, Mark tab, click Clear all marks.

Remove Duplicate Lines

  • Method 1
    • Edit --> Line Operations --> Remove Duplicate Lines
    • Edit --> Line Operations --> Sort Lines Lexicographically Ascending (this is optional but makes them easier to read.)
  • Method 2
    • Edit --> Line Operations --> Sort Lines Lexicographically Ascending
    • Edit --> Line Operations --> Remove Consecutive Duplicate Lines

Delete the first part of a line

Find:   ^.*YourCodeHere
Replace:  YourCodeHere

or

Find:   ^.*?YourCodeHere
Replace:  YourCodeHere

Line Endings (EOL / CR / LF)

A collection of hints and tips of this topic.

  • The default Notepad++ font `Courier New` does not support a lot of Unicode characters such as the CRLF character.
  • Notepad++ End of Line: How to Change It and Why - Notepad++ end of line settings: Learn how to change the end of line character in Notepad++. This article includes step-by-step instructions and screenshots.

To View Line Endings in Notepad++:

  1. Go to View > Show Symbol > Show End of Line
    • CR LF for Windows line endings
    • LF for Unix-style
    • CR for old Mac-style
    • This will display actual newline characters:
    • These aren’t shown as ↵, but they indicate the same thing.

If you want ↵ to Show where Newlines Occur:

Notepad++ supports showing newlines as CR/LF/CRLF but does not support showing newlines as ↵, however you can simulate it:

  1. Use Find and Replace (Ctrl+H)
    • Set Search Mode to Extended
    • Find: \r\n or \n
    • Replace with: ↵\r\n (this will visually insert ↵ at line breaks)
    • Be careful—this alters the file.

Regex

Themes

Misc

How to add a TPL file to Notepad++ as a PHP file

  1. Click Settings / Style Configurator
  2. Select PHP in the left list box
  3. At the bottom of the list boxes, you'll see User Ext:. Enter tpl into that box
  4. Click Save & Close

Now Notepad++ will recognise .tpl files as PHP files. When you open the next one, it will contain syntax highlighting.

A question mark or a box is shown instead of the EOL character?

  • I am using the Fira Code here for this example but this will apply to all fonts where this occurs
  • The EOL character is shown when using the DejaVu Mono font.
  • The question mark character is shown is when the required glyph does not exist in the current font.
  • When copying text, the glyph references are copied not the characters, so if a character does not show correctly in one application this does not mean this will be the same in another.

The issue you're encountering is due to the Fira Code font not supporting the ↵ character (Unicode U+21B5: "DOWNWARDS ARROW WITH CORNER LEFTWARDS").

Here's a breakdown of what's going on:

  • ✅ DejaVu Mono
    • DejaVu Mono is a very complete font that includes a wide range of Unicode symbols, including many technical and typographic ones.
    • It supports the ↵ character, so it displays correctly.
  • ❌ Fira Code
    • Fira Code is a monospaced programming font known for its ligatures (like turning != into a neat symbol), but it has a more limited set of Unicode glyphs compared to DejaVu Mono.
    • It does not include a glyph for ↵, so Notepad++ shows a replacement character (typically a question mark or a box), indicating a missing glyph.

Solutions or Workarounds

  1. Use a different symbol:
    • You can try using a visually similar symbol that Fira Code does support (e.g., ↩, ↓, or ⏎), though it’s not the same.
  2. Use a different font:
    • If Unicode coverage is important and you're dealing with many special characters, consider using a font like:
      • JetBrains Mono
      • Cascadia Code
      • DejaVu Mono
      • DejaVu Sans
      • Segoe UI Symbol
      • Arial Unicode MS
    • Use a custom build of Fira Code with extended Unicode coverage (more advanced).
  3. Use a different editor:
    • Switch to a different editor that supports font fallback (like VS Code, Sublime Text, etc.).

To Verify a Character is supported in a font

You can test if Fira Code supports a character by:

  • Checking it in a font viewer (like BabelMap or FontDrop).
  • Viewing it in a browser with Fira Code explicitly set as the font.
  • Set your editor to use the required font, then copy and paste in the character from somewhere it is shown correctly.

Why ↵ works in VS Code but Not in Notepad++

VS Code and Notepad++ handle fonts very differently, particularly when it comes to missing glyphs:

  • ✅ VS Code: Font Fallback Support
    • VS Code uses Electron/Chromium, which supports font fallback.
    • When Fira Code doesn't include a glyph (like ↵), VS Code will automatically fall back to another installed font that does support it (e.g., DejaVu Sans Mono or Segoe UI Symbol).
    • So even though Fira Code doesn’t include ↵, VS Code silently substitutes it from another font.
  • ❌ Notepad++: No Font Fallback
    • Notepad++ uses the Windows GDI or Scintilla rendering engine, which lacks proper font fallback.
    • If the selected font (Fira Code) doesn’t include a glyph, Notepad++ simply shows a replacement character (usually a box or question mark), because it won’t switch to another font.

✅ Summary

App Font Used ↵ Glyph Supported by Font? Font Fallback? Result
VS Code Fira Code ❌ ✅ Yes ✔️ Displays ↵
Notepad++ Fira Code ❌ ❌ No ❌ Question mark

Additional Fonts

 

Published in Applications
Read more...
Saturday, 24 May 2025 09:02

Enable long file paths in Windows

Window should just enable this by default but it doesn't even in windows 11.

TL;DR

Methods

The tutorials in the links below will give you a clearer understanding of how and why to perform this upgrade but this is an overview of the methods I found

  1. Registry 
    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem
  2. PowerShell (to change the registry) 
    Set-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem' -Name LongPathsEnabled -Type DWord -Value 1
  3. Group Policy Manager (Windows Pro only) 
    Computer Configuration > Administrative Templates > System > Filesystem > Enable Win32 long paths

Notes

  • Some legacy apps, in particular Win32 will have issues with long file paths.
  • Feature was brought in with Windows 1607
  • Not on by default, including in Windows 11
  • File explorer in Windows 11 will read long paths correctly but if you are over the 260 character limit it will not allow you to edit or create them (as of Windows 11 24H2)
  • Apps have to support long file paths, just enabling the setting is not enough.
  • Using a 3rd party file manager will get around the file explorer limit as long as LongPathsEnabled is enabled

Links

  • Background
  • Tutorials
    • Maximum Path Length Limitation - Win32 apps | Microsoft Learn - Starting in Windows 10, version 1607, MAX_PATH limitations have been removed from many common Win32 file and directory functions. However, your app must opt-in to support the new behavior.
    • How to Make Windows 10 Accept File Paths Over 260 Characters | How-To Geek - Want a small novel for a file path? We've got you covered.
      • Before Windows 95, file names could only be 8 characters long, but now Windows has a 260 character limit for the full path of a file.
      • The Windows 10 Anniversary Update allowed users to abandon the 260 character limit, but some older 32-bit applications may not support longer paths.
      • Home users can remove the path limit by editing the registry, while Pro and Enterprise users can use the Local Group Policy Editor to disable the limit.
      • The maximum path limit may not be something you've ever run into, but for some people, it can certainly be the occasional frustration. Windows 10 has finally added the ability to remove that limit. You just have to make a quick change to the Registry or Group Policy to make it happen.
      • Download the registry files.
    • Windows 10 Allows File Paths Longer Than 260 Characters (With a Registry Hack) | Lifehacker - Ever since Windows 95, Microsoft has only allowed file paths up to 260 characters (which, to be fair, was much nicer than the 8 character limit previously). Now, with a registry tweak, you can exceed that amount in Windows 10.
    • Maximum Path Length Limitation - Block repositories | Veeam Backup & Replication Best Practice Guide
      • Best practice from the field for Veeam Backup & Replication
      • Regardless of NTFS or ReFS, Windows operating systems used to have a limit of 260 characters as the maximum length of a file’s path name. Although this limitation has been removed in newer versions starting with Windows 10 version 1607, this feature is still disabled by default within the Windows API (see Maximum Path Length Limitation).
      • Veeam is capable of handling long path names, but it needs to be enabled in the OS first. It is recommended to enable the LongPathsEnabled setting on all Windows servers hosting Veeam roles.
      • Refer to the Microsoft documentation for more details on how to change this value via the registry, PowerShell or Group Policy.
  • Windows Explorer "does not support long file paaths
    • Use a 3rd party app to overcome this. Obviously this needs to support long file paths
    • STILL no file paths longer than 260 characters in Win Explorer, in 2023? - Super User
      • Your only option would be to use an alternative File Explorer that supports Long Paths. Examples are OneCommander, Total Commander, and 7-Zip. (A nicer option that seems to support it, and can act as a Windows Explorer replacement with a similar layout, but isn't free, is the Files App.)

dddd

Published in Windows General
Read more...
Sunday, 11 May 2025 09:19

Line endings when working with Git for Windows and Visual Studio Code

So you are installing `Git for Windows` and you are asked how to handle line endings, what option should I pick and why? I will answer this question here so you know what to choose and why rather than just following a YouTube video than never explains anything except just click next because this is what I use.

The information here applies to handling EOL in Git and probably applies to people whoe's OS is not Windows such as MacOS and Linux.

Summaries

Where I can change line endings when using Git for Windows and VSCode? (each overrides the last)

Settings precedence - User and workspace settings | Visual Studio Code

  1. VSCode (User settings) (%APPDATA%\Code\User\settings.json)
    • Action: Set EOL for new files.
    • Scope: All Workspaces.
    • Additional:
      • Command Palette ( F1 | Ctrl+Shift+P) --> Preferences: Open User Settings --> `Files: Eol`: LF
      • Settings --> Text Editor --> Files --> Eol: LF
      • Profiles replace the `User settings`, if loaded.
      • These settings are only per user which means they are local and are not shared.
  2. VSCode (Workspace settings) (.vscode/settings.json)
    • Action: Set EOL for new files.
    • Scope: Workspace.
    • Additional:
      • These settings can be commited to a repo and thus can be shared between colleagues and the wider world.
  3. VSCode (Multi-Root Workspace / Workspace Folder setting) (.vscode/.code-workspace)
    • Action: Set EOL for new files.
    • Scope: Specified Workspace Folders.
    • Additional:
      • Settings for a particular folder can be saved in the .code-workspace.
      • These settings can be commited to a repo and thus can be shared.
  4. VSCode (Multi-Root Workspace) ([Workspace Folder]/.vscode/settings.json)
    • Action: Set EOL for new files.
    • Scope: Workspace Folder
    • Additional:
      • These settings can be commited to a repo and thus can be shared.
  5. Git (Repository) (.editorconfig)
    • Action: Set EOL for new files.
    • Scope: Git repository.
    • Additional:
      • Visual Studio = Full native support, VS will check your code on every code change.
      • VSCode = Partial support is added by extensions.
  6. VSCode (Extensions) - Examples below:
    1. Auto Convert End Of Line - Visual Studio Marketplace
      • Action: Change the EOL for all text files, on save
      • Scope: Workspace
    2. Prettier - Code formatter - Visual Studio Marketplace
      • Action: Change the EOL for all files, as specified (see below) on save
      • Scope: Workspace
      • Additional:
        • You can use VS Code settings to configure prettier. Settings will be read from (listed by priority):
        • NOTE: If any local configuration file is present (i.e. .prettierrc) the VS Code settings will NOT be used.
        • The VS Code settings are meant to be used as a fallback and are generally intended only for use on non-project files.
        • It is recommended that you always include a prettier configuration file in your project specifying all settings for your project.
        • If options.editorconfig is true and an .editorconfig file is in your project, Prettier will parse it and convert its properties to the corresponding Prettier configuration. This configuration will be overridden by .prettierrc, etc. Currently, the following EditorConfig properties are supported:
          • end_of_line
          • indent_style
          • indent_size
          • tab_width
          • max_line_length
        • End of Line - Options · Prettier - Prettier ships with a handful of format options.
        • Unlocking Prettier: Mastering End of Line Configuration | runebook.dev
          • Inconsistent EOL characters can lead to unnecessary Git conflicts, making it harder to track changes and collaborate effectively.
          • Prettier aims to enforce consistent code style across a project or team. This includes standardizing how line endings are represented.
          • This article covers .prettierrc and .gtiattributes
    3. ESLint - Linter - Visual Studio Marketplace
      • Action: Change the EOL for all files, as specified (see below) on save
      • Scope: Workspace
    4. EditorConfig for VS Code - Visual Studio Marketplace
      • Action: Change the EOL for all files, as specified in .editorconfig, on save
      • Scope: Workspace
      • Additional:
        • Supported Properties
          • indent_style
          • indent_size
          • tab_width
          • end_of_line (on save)
          • insert_final_newline (on save)
          • trim_trailing_whitespace (on save)
  7. Git (Repository) (.gitattributes)
    • Action: Change the EOL for all files
    • Scope: Git repository, but only on files as specified in the .gitattributes
    • Additional:
      • Applied when files are checked-in (submitted) to this Git repo.
  8. Git (Global Config / Git for Windows) (.gitconfig)
    • Action: Change the EOL for all files
    • Scope: All local Git repositories.
    • Additional:
      • Occurs when files are submitted or retrieved from any Git repo.
      • Only core.autocrf=true performs a conversion on EOLs during on Check-in and Checkout, all other settings are Checkin-in only.

 

How to enforce LF EOL (Windows --> VSCode --> Git)

Making the following changes to your Editor and Repo will ensure the best outcome

  • VSCode (User settings) (%APPDATA%\Code\User\settings.json)
    • Scope:
      • Set EOL LF for new files, for the User.
    • Actions:
      • Enable LF as the default EOL for the current VSCode User
        • Command Palette --> Preferences: Open User Settings --> `file end of line` --> `Files: Eol`: \n
          • It will indicate LF at the bottom of the dropdown, which is fine as this is is just a note.
  • VSCode (Workspace settings) (.vscode/settings.json)
    • Scope:
      • Set EOL LF for new files, for the Workspace.
    • Actions:
      • Enable LF as the default EOL for the VSCode Workspace
        • .vscode/settings.json 
          {
  "files.eol": "\n"
}

          Command Palette --> Preferences: Open Workspace Settings --> `file end of line` --> `Files: Eol`: \n

  • Git (Repository) (.editorconfig)
    • Scope:
      • Set EOL LF for new files, for the Git Repository.
      • Some VSCode extensions will use the EOL and apply upon save.
    • Actions:
      • Enable .editorconfig support (Visual Studio does not need this)
        • Install EditorConfig for VS Code
        • Not required if you use Prettier as this has supports, but you will need to make sure it is configured to respect .editorconfig however .prettierrc settings will always override .editorconfig values.
        • .editorconfig sets the EOL for new files but these VSCode extensions set the EOL on save.
      • Create .editorconfig 
        # EditorConfig is awesome: https://EditorConfig.org

# top-most EditorConfig file
root = true

# Default Behaviour
[*]
end_of_line = lf
  • VScode (Extensions) (.prettierrc) (Prettier)
    • Scope:
      • Can use .editorconfig or it's own settings, Set EOL LF for new files, for the Git Repository.
      • Some VSCode extensions will use the EOL and apply upon save.
    • Actions:
      • Create .prettierrc (only needed if using Prettier) 
        {
  "endOfLine": "lf"
}
  • VScode (Extensions) (eslint.config.js) (ESLint)
    • SCope
      • Can use .editorconfig or it's own settings, Set EOL LF for new files, for the Git Repository.
      • Some VSCode extensions will use the EOL and apply upon save.
    • Actions:
      • Create eslint.config.js (only needed if using ESLint) 
        import { defineConfig } from "eslint/config";

export default defineConfig([
  // Default Line Ending
  {
    files: ['**/*.*'],
    rules: {
      'linebreak-style': ['error', 'unix'],
    },
  },

  // Force batch scripts to always use CRLF line endings
  {
    files: ['**/*.[bB][aA][tT]', '**/*.[cC][mM][dD]'],
    rules: {
      'linebreak-style': ['error', 'windows'],
    },
  },

  // Force bash scripts to always use LF line endings
  {
    files: ['**/*.sh'],
    rules: {
      'linebreak-style': ['error', 'unix'],
    },
  },
]);
  • Git (Repository) (.gitattributes)
    • Scope
      • Enforce conversion of EOL to LF.
    • Actions:
      • Create .gitattributes (optional) 
        # Default Line Ending
* text=auto eol=lf
        • This will change all text file's EOL to LF when checking them into the repository, but not when you check them out in to your working tree (i.e. real files).
        • This is a fallback to enforce EOL before committing and should not be needed if everything in your repo is setup correctly.
        • Using this setting (* text=auto eol=lf) on it's own can cause corruption in some case, so a more granular .gitattributes configuration should be usedas Git can make mistakes with line endings, e.g. mixed EOL files and those with binary blobs
        • Use and this is a base reference and also consult further notes on this page.
        • * text=auto eol=lf is the same as * text=auto
        • This rule is doing the same for this repository as core.autocrf=input would do for the whole local Git instance.

 

EOL Handling Summary

  • General
    • DO NOT use core.autocrlf to normalise your line endings
      • This can lead to file corruption (i.e. embedded binary blobs in a text file)
      • You cannot assume everyone has set their Git development environment the same.
      • These settings would be global and thus a local repository would not have control and nor could you mandate changes with your editor.
    • Use .gitattributes and .editorconfig to configure line normalisation
      • These work on a Per-Repository basis.
      • This will allow you granular control of EOL.
      • Supports files with different EOL.
      • Support an individual file with mixed EOL.
      • The repository can define the EOL handling rather than individual user environments.
      • This is the professional way to setup a repository.
  • .git
    • Repo (check-in) = Commit = Submitted to Git
    • Working Tree (checkout) = The real files on your PC
  • .gitconfig
    • core.autocrlf=true
      • Converts text file's EOL to LF on check-in to the repo from the working tree.
      • Converts text file's EOL to CRLF on checkout from the repo to the working tree.
      • Tends to be recommended setting for Windows developers since it guarantees LF in the remote copy of your code while allowing you to use CRLF in your working tree for full compatibility with Windows editors and file formats.
      • This setting only needs be used in a Window environment, but can work in other OS.
    • core.autocrlf=input
      • Converts text file's EOL to LF on check-in to the repo from the working tree.
      • Leaves the working tree untouched.
      • This would be the same as: * text=auto eol=lf and * text=auto, but applied globally.
  • .gitattributes
    • Controls how EOL are handled when checking-in files to the Git repository.
    • Local files (Working Tree) and their EOL are unaltered by these settings.
    • Conflicts might occur when checking out files, but these can be resolved locally without poisoning the repo.
    • * text=auto eol=lf, * text=auto and core.autocrlf=input all do the same thing except core.autocrlf=input works at the Git server level and not on a per repository basis.
    • (untested) This file will override the core.autocrlf settings for specified files in this repository if an appropriate rule exists.
    • Using this file ensures a consistent behavior for all users, regardless of their Git settings and environment.
  • .editorconfig
    • An open, editor agnostic schema.
    • Defines the EOL for all new files.
    • Some VSCode extensions use these settings on save, so these settings can work on new and created files.
  • VSCode (User Settings)
    • Configuring EOL here will only apply to new files.
    • You should change the default EOL to LF as a fallback precaution (unless you are working on Windows only environment).
    • These settings are only per user which means they are local and are not shared.
  • VSCode (Workspace settings)
    • Configuring EOL here will only apply to new files.
    • You should change the default EOL to LF as a fallback precaution (unless you are working on Windows only environment).
    • These settings can be commited to a repo and thus can be shared.

Git for Windows - Installer - Questions

Checkout as-is, commit as-is (Git for Windows) (core.autocrlf = false)

Why should I choose this option, here is a simple Pros vs Cons:

  • Pro
    • All modern editors can handle all EOL types and does not need Git to do any conversions.
    • Your code will be exactly the same in the local Git as well as the remote repository.
    • Your code will not get corrupted because of the EOL conversions
    • You can use a mix of files with different EOL.
    • You can submit single files with mixed EOL (currently VSCode does not support mixed EOL).
    • You can submit files with any EOL (CRLF / CR / LF).
    • Can granular control which files have their EOL translated by Git at the repository level using .gitattributes.
    • EOL specified by .editorconfig will be honoured and kept when the files are submitted to the repository.
  • Con
    • Requires the editor and repository to be setup correctly to prevent EOL inconsistencies amongst programmers.

Checkout Windows-style, commit Unix-style line endings (Git for Windows) (core.autocrlf = true)

Why should I choose this option, here is a simple Pros vs Cons:

  • Pro
    • If your files in a project have different EOL, this could ensure compatibility with minimal input from the user.
      • = You should set your development environment up better.
    • Good for people who work across different OS and platforms to ensure the repository always has the correct line endings.
      • = You should set your development environment up better.
  • Con
    • Can cause corruption in your code e.g. binary blob.
    • All EOL will be changed, meaning that if there are mixed EOL, then this difference will be lost whether that be in a single file or files with different EOL.

Bulk Changing EOL

So you already have a load of files with CRLF and now you need them all to be LF before committing them to Git.

  • VSCode
    • Change All End Of Line Sequence
      • Runs the built-in 'Change End Of Line Sequence' on all files of a specified type, leaving them open and unsaved.
    • Auto Convert End Of Line
      • Automatically convert files with configured end-of-line sequence when they are opened.
      • Every time a text files is opened, it converts all the EOL sequences into the specific one from the files.eol configuration, and displays an auto-dismissed notification. If no conversion is needed, or if files.eol is configured to be "auto", this extension does nothing.
  • Windows
  • Linux
    • Converting between Windows and Linux line breaks | Commandline Wizardry 
      sed -i 's/$/\r/' fileLinux.txt
find -type f -exec sed -i 's/$/\r/' '{}' \;
tr -d '\r' < fileWindows.txt > fileLinux.txt
sed -i 's/\r//' fileWindows.txt
    • Dos2unix Command on Linux with Examples - idroot - Learn how to effectively use the 'dos2unix' command on Linux to ensure seamless file compatibility.
    • Dos2unix: The Linux File Conversion Command Explained
      • Ever found yourself grappling with line ending issues when transferring text files between Windows and Unix/Linux systems? You're not alone. You’re not alone. Many developers find themselves in this predicament, but there’s a tool that can help you navigate these choppy waters.
      • In this guide, we’ll walk you through the process of using the dos2unix command in Linux, from the basics to more advanced techniques. We’ll cover everything from simple file conversions to handling multiple files, as well as alternative approaches and troubleshooting common issues.
      • The dos2unix command is a straightforward yet powerful tool in the Linux arsenal. Its primary role is to convert text files from DOS/MAC to Unix format, which is vital when transferring files between different operating systems.
    • Find Out Text File Line Endings - LF or CLRF - ShellHacks
      • How to find out the line endings in a text file and how to convert the Unix/Linux newline characters ("n" or "LF") to DOS/Windows ("rn" or "CRLF").
      • Change Newline Characters 
        ## To convert the newline characters from DOS/Windows to Unix/Linux:
$ dos2unix <filename>

## To convert the newline characters from Unix/Linux to DOS/Windows:
$ unix2dos <filename>
    • Conversion between newline formats - Newline | Wikipedia - Many Linux examples here.
  • Python
  • NodeJS

Links / Research

History of CRLF and LF

Handling Line Conversions articles

  • Configuring Git to handle line endings - GitHub Docs
  • Harmonizing Line Endings Across Windows and WSL | xakpc.info - When using a Git repository across Windows and Windows Subsystem for Linux (WSL), you may encounter unexpected issues with line endings. This article explains why these issues occur and how to resolve them effectively.
  • Guide to line ending handling in Git · GitHub - This guide serves to clear up how Git handles line endings, how to avoid problems with them, how to fix problems if they come up, and some great tools for dealing with line endings in general.
  • Git LF to CRLF: Mastering Line Endings Simply - Discover why git lf will be replaced by crlf and how to adapt your workflow. This concise guide simplifies the transition for smooth version control.
    • Consistent Configuration: Ensure that every team member has set their `core.autocrlf` appropriately for their operating system.
    • Use of .gitattributes: Proper configuration of the `.gitattributes` file can minimize discrepancies; this ensures everyone follows the same rules for line endings.
  • Resolving Git Line Ending Issues Once and For All | TheLinuxCode - In this comprehensive 2500+ word guide, you‘ll gain a deeper understanding of controlling Git line endings plus actionable solutions to banish newline woes.
  • git - How to change line-ending settings - Stack Overflow
    • Global Configuration
      • In Linux/OSX 
        git config --global core.autocrlf input
        This will fix any CRLF to LF when you commit.
      • In Windows 
        git config --global core.autocrlf true
        This will make sure that, when you checkout in windows, all LF will be converted to CRLF.
    • .gitattributes File
      • It is a good idea to keep a .gitattributes file as we don't want to expect everyone in our team to set their own config. This file should be placed in the repository root and, if it exists, git will respect it.
  • * text=auto in .gitattributes file - DEV Community - In this article, we analyse the .gitattributes from the react/.gitattributes. When you are part of a team, you and your colleagues often work on projects across different operating systems and development environments. As a result, file formatting issues, especially related to line endings, can arise.

Pro (Checkout as-is, commit as-is)

  • (.gitattributes)
    • How to configure Git line endings correctly | LabEx (Read this)
      • Master Git line ending configurations for seamless cross-platform development, prevent file conflicts, and ensure consistent code formatting across different operating systems.
      • This is a detail article with relevant diagrams and explores all aspects of this topic.
      • Using .gitattributes for explicit control and set EOL in VSCode to be \n
      • .gitattributes Example 
        ## Specify precise line ending behaviors
* text=auto
*.sh text eol=lf
*.bat text eol=crlf
*.txt text eol=lf
    • How To: Handle Line Endings with Git | Shane Bart
      • Learn how to handle line endings when using git across various operating system platforms
      • Option 1: ‘core.autocrlf’ git config (legacy approach)
        • Whilst this solution will work well in a single developer environment. It is not recommended in a multi-developer environment, as it relies on all developers in the team to explicitly configure their git configuration, which may not always happen.
      • Option 2: Using .gitattributes (Recommended)
        • This enables you to capture all required configuration in source control, rather than needing to rely on developers within your team to explicitly configure git.
      • Also, try to stay away from using both these approaches together as it is more than likely to create confusion and lead to issues.
    • .gitattributes Best Practices - Muhammad Rehan Saeed (Read this)
      • Git can actually be configured to automatically handle line endings using a setting called autocrlf. This automatically changes the line endings in files depending on the operating system. However, you shouldn't rely on people having correctly configured Git installations.
      • Covers many aspects of this file and how useful it is.
      • A complete .gitattributes file
      • Only the first line is strictly necessary. It hard codes the line endings for Windows cmd and batch scripts to CRLF and bash scripts to be LF, so that they can be executed via a file share. It's a practice I picked up from the corefx repository.
      • From comments
        • Most of the programming editors these days support working with either line ending, regardless of the platform they run on, and most other tools (compilers etc..) are line-ending-agnostic.
        • I've found I've had more problems caused by git trying to be helpful and converting my line endings for me, than the other way around. So I'd say best practice is to use * -text in the .gitattributes and add .editorconfig that enforces the projects preferred line endings. An added benefit is that you can also specify file encodings in your .editorconfig file.
      • .gitattributes Example 
        ###############################
# Git Line Endings            #
###############################

# Set default behaviour to automatically normalize line endings.
* text=auto

# Force batch scripts to always use CRLF line endings so that if a repo is accessed
# in Windows via a file share from Linux, the scripts will work.
*.{cmd,[cC][mM][dD]} text eol=crlf
*.{bat,[bB][aA][tT]} text eol=crlf

# Force bash scripts to always use LF line endings so that if a repo is accessed
# in Unix via a file share from Windows, the scripts will work.
*.sh text eol=lf

###############################
# Git Large File System (LFS) #
###############################

# Archives
*.7z filter=lfs diff=lfs merge=lfs -text
*.br filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text

# Documents
*.pdf filter=lfs diff=lfs merge=lfs -text

# Images
*.gif filter=lfs diff=lfs merge=lfs -text
*.ico filter=lfs diff=lfs merge=lfs -text
*.jpg filter=lfs diff=lfs merge=lfs -text
*.pdf filter=lfs diff=lfs merge=lfs -text
*.png filter=lfs diff=lfs merge=lfs -text
*.psd filter=lfs diff=lfs merge=lfs -text
*.webp filter=lfs diff=lfs merge=lfs -text

# Fonts
*.woff2 filter=lfs diff=lfs merge=lfs -text

# Other
*.exe filter=lfs diff=lfs merge=lfs -text
    • Git for Windows: Line Endings | Edward Thomson
      • The key to dealing with line endings is to make sure your configuration is committed to the repository, using .gitattributes. 
      • Checkout Windows-style, commit Unix-style line endings: This is why one of the most common symptoms of a line ending configuration problem is seeing “phantom changes”: running git status tells you that you’ve changed a file, but running git diff doesn’t show you any changes.
    • Git Handling Line Endings (CRLF) | git-cookbook
      • This covers both .gitconfig and .gitattributes with good examples
      • CRLF conversion bears a slight chance of corrupting data. When core.autocrlf is enabled, Git will convert CRLF to LF during commit and LF to CRLF during checkout. A file that contains a mixture of LF and CRLF before the commit cannot be recreated by Git. For text files this is the right thing to do: it corrects line endings such that we have only LF line endings in the repository. But for binary files that are accidentally classified as text the conversion can corrupt data.
      • Unfortunately, the desired effect of cleaning up text files with mixed line endings and the undesired effect of corrupting binary files cannot be distinguished. In both cases CRLFs are removed in an irreversible way. For text files this is the right thing to do because CRLFs are line endings, while for binary files converting CRLFs corrupts data.
      • core.safecrlf is basically just a way to make sure you don’t irreversably perform CRLF conversion on a binary file.
      • Optionally, you can configure a .gitattributes file to manage how Git reads line endings in a specific repository. When you commit this file to a repository, it overrides the core.autocrlf setting for all repository contributors. This ensures consistent behavior for all users, regardless of their Git settings and environment.
      • Some good examples.
      • core.eol - Sets the line ending type to use in the working directory for files that are marked as text. Alternatives are lf, crlf and native, which uses the platform’s native line ending. The default value is native. 
        git config core.eol lf      # LF
git config core.eol crlf    # CRLF
git config core.eol native  # platform (default)
      • .gitattibutes example 
        # Set the default behavior, in case people don't have core.autocrlf set.
* text=auto

# Explicitly declare text files you want to always be normalized and converted
# to native line endings on checkout.
*.c text
*.h text

# Declare files that will always have CRLF line endings on checkout.
*.sln text eol=crlf

# Denote all files that are truly binary and should not be modified.
*.png binary
*.jpg binary
  • (.gitattributes) (.editorconfig)
    • CRLF vs. LF: Normalizing Line Endings in Git | Aleksandr Hovhannisyan (Read this)
      • Line endings can differ from one OS to another. Learn the history behind CRLF and LF line endings and how to enforce line endings in Git.
        • The typical advice is to configure your local Git to handle line ending conversions for you. For the sake of comprehensiveness, we’ll look at how that can be done in this article, but it isn’t ideal if you’re on a large team of developers.
          • The core.autocrlf setting isn’t the ideal approach for configuring line endings in a project.
        • A better solution is to add a .gitattributes file to your repo so you can enforce line endings consistently in your codebase regardless of what operating systems your developers are using.
        • A .gitattributes file is technically all that you need to enforce the line endings in the remote copy of your code. However, as we just saw, you may still see CRLF line endings on Windows locally because .gitattributes doesn’t tell Git to change the working copies of your files. Fortunately, we can take things a step further with an .editorconfig file; this is an editor-agnostic project that aims to create a standardized format for customizing the behavior of any given text editor. Lots of text editors (including VS Code) support and automatically read this file if it’s present.
        • Gives the history of these control characters and the specific differences.
        • Configuring Line Endings in Git with core.autocrlf
          • By default, core.autocrlf is set to false on a fresh install of Git, meaning Git won’t perform any line ending normalization. Instead, Git will defer to the core.eol setting to decide what line endings should be used; core.eol defaults to native, which means it depends on the OS you’re using. That’s not ideal because it means that CRLF may make its way into your code base from Windows devs.
          • Both of these options enable automatic line ending normalization for text files, with one minor difference: core.autocrlf=true converts files to CRLF on checkout from the repo to the working tree, while core.autocrlf=input leaves the working tree untouched.
          • For this reason, core.autocrlf=true tends to be recommended setting for Windows developers since it guarantees LF in the remote copy of your code while allowing you to use CRLF in your working tree for full compatibility with Windows editors and file formats.
        • A Simple .gitattributes Config
          • The following .gitattributes config normalizes line endings to LF for all text files checked into your repo while leaving local line endings untouched in the working tree:
          • More specifically, the text=auto option tells Git to only normalize line endings to LF for text files while leaving binary files (images, fonts, etc.) untouched. This distinction is important—we don’t want to corrupt binary files by modifying their line endings.
          • After committing the .gitattributes file, your changes won’t take effect immediately for files checked into Git prior to the addition of .gitattributes. To force an update, you can use the following command since Git 2.16:
        • Many Git commands are shown and explained.
        • Summary
          • That was a lot to take in, but hopefully you now have a better understanding of the whole CRLF vs. LF debate and why this causes so many problems for teams that use a mixture of Windows and other operating systems. Whereas Windows follows the original convention of a carriage return plus a line feed (CRLF) for line endings, operating systems like Linux and Mac use only the line feed (LF) character.
          • The history of these two control characters dates back to the era of the typewriter. While this tends to cause problems with software like Git, you can specify settings at the repo level with a .gitattributes file to normalize your line endings regardless of what operating systems your developers are using.
          • You can also optionally add an .editorconfig file to ensure that new files are always created with LF line endings, even on Windows.
        • Comments
          • Using a .gitattributes file is indeed the best practice, but if you are working on a personal repository, it shouldn’t matter. In fact, autocrlf=true wouldn’t cause any problems even if you were working with other Linux/Mac developers since the remote code would always have LF line endings anyway. The main reason why autocrlf is not useful is because you have to keep setting it up every time you get a new Windows machine (since line ending normalization is turned off by default in Git). If you forget to set it up, you’ll re-introduce CRLF into the remote code. From my article:
    • Normalizing Line Endings in Git: CRLF vs. LF - MyBlueLinux.com
      • If you’ve ever worked on a project where developers use different operating systems, you know that line endings can be a peculiar source of frustration.
      • The typical advice is to configure your local Git to handle line ending conversions for you. For the sake of comprehensiveness, we’ll look at how that can be done in this article, but it isn’t ideal if you’re on a large team of developers.
      • A better solution is to add a .gitattributes file to your repo so you can enforce line endings consistently in your codebase regardless of what operating systems your developers are using.
      • We’ll briefly review the history behind line endings on Windows and Unix so we can understand why this issue exists in the first place.
      • By default, core.autocrlf is set to false on a fresh install of Git, meaning Git won’t perform any line ending normalization. Instead, Git will defer to the core.eol setting to decide what line endings should be used; core.eol defaults to native, which means it depends on the OS you’re using. That’s not ideal because it means that CRLF may make its way into your code base from Windows devs.
      • That leaves us with two options if we decide to configure Git locally: core.autocrlf=true and core.autocrlf=input. The line endings for these options are summarized below.
        • core.autocrlf=true / LF / CRLF
        • core.autocrlf=input / LF / original (usually LF, or CRLF if you're viewing a file you created on Windows)
        • Both of these options enable automatic line ending normalization for text files, with one minor difference: core.autocrlf=true converts files to CRLF on checkout from the repo to the working tree, while core.autocrlf=input leaves the working tree untouched.
      • core.autocrlf=true / * text=auto eol=crlf / LF / CRLF
      • core.autocrlf=input / * text=auto / LF / original (LF or CRLF)
  • (.editorconfig)
    • Using the .editorconfig in VS2019 and VSCode | ASP.NET Hacker 
      • In the backend developer team of the YOO we are currently discussing coding style guidelines and ways to enforce them. Since we are developer with different mindsets and backgrounds, we need to find a way to enforce the rules in a way that works in different editors too.
      • The .editorconfig is a text file that overrides the settings of the editor of your choice.
      • Almost every code editor has settings to style the code in a way you like, or the way your team likes. If this editor supports the .editorconfig you are able to override this settings with a simple text file that usually is checked in with your source code and available for all developers who work on those sources.
      • If VS2019 finds an .editorconfig it will use it immediately and it will check your code on every code change.
    • Windows Git Users: Don't Convert Line Endings - With conversion enabled, the checked-out files are in a different state than their source repository. This can lead to strange errors where build artifacts or files transferred to other systems are subtly different than those generated from the unmodified source.

Git and Repository EOL Configuration

  • .gitconfig (Global settings)
  • .gitattributes
    • Configuring Git to handle line endings - GitHub Docs - To avoid problems in your diffs, you can configure Git to properly handle line endings.
    • gitattributes | Documentation | git-scm.com - List the relevant switches and values.
    • text - gitattributes | Documentation | git-scm.co
      • This attribute marks the path as a text file, which enables end-of-line conversion: When a matching file is added to the index, the file’s line endings are normalized to LF in the index. Conversely, when the file is copied from the index to the working directory, its line endings may be converted from LF to CRLF depending on the eol attribute, the Git config, and the platform (see explanation of eol below).
    • 8.2 Customizing Git - Git Attributes | Documentation | git-scm.com
      • Using attributes, you can do things like specify separate merge strategies for individual files or directories in your project, tell Git how to diff non-text files, or have Git filter content before you check it into or out of Git.
      • In this section, you’ll learn about some of the attributes you can set on your paths in your Git project and see a few examples of using this feature in practice.
    • Git .gitattributes | W3 Schools
      • The .gitattributes file is a special file that tells Git how to handle specific files in your repository.
      • It controls things like line endings, file types, merge behavior, custom diff tools, and more.
      • Everyone on your team gets the same settings because this file is versioned with your project.
      • Put .gitattributes in subfolders for rules that only apply there.
    • gittattributes force all text files to lf - Bing Search
      • To ensure all text files in your Git repository use LF (Line Feed) as the line-ending format, you can configure a .gitattributes file. Here's how you can do it:
      • Steps:
        • Create or edit a .gitattributes file in the root of your repository.
        • Add the following line to enforce LF for all text files: 
          * text=auto eol=lf
      • Explanation:
        • *: Matches all files in the repository.
        • text=auto: Tells Git to automatically detect text files.
        • eol=lf: Ensures that line endings are normalized to LF when files are checked out.
      • Additional Notes:
        • If you already have files with CRLF endings in your repository, you may need to normalize them. Run the following commands after setting up .gitattributes: 
          git add --renormalize .
git commit -m "Normalize line endings to LF"
        • This ensures consistency across different operating systems and prevents issues caused by mismatched line endings.
    • Why does .gitattributes not override core.autocrlf configuration on Linux? - Stack Overflow
      • So if you want to use attributes to control the line ending you get in the work tree, you need to set the eol attribute (not merely the text attributes).
      • So, to answer your question: no, what you've described is not a bug.
  • .editorconfig

VSCode EOL Configuration

 

Published in Programming
Read more...
Monday, 05 May 2025 18:10

My Passkeys Notes

to be continued

Published in Web Security
Read more...
Sunday, 04 May 2025 07:19

My Netbeans Notes

Add a custom file extension to Netbeans

I am trying to figure out how to add a custom file type to be recognised by Netbeans.

I am editing .tpl files and I would like them to be recognised as PHP/HTML files.

Solution

  1. Open preferences.
  2. Select Miscellaneous tab.
  3. Select Files sub-tab thing.
  4. Click on New file extension and enter tpl.
  5. Select the mime type.
  6. Click ok. Done!

Coloured background in tpl cannot be turned off

  1. go to Tools -> Options -> Fonts&Colors
  2. choose language 'Smarty Templates'
  3. choose Category 'Background'
  4. you can change colour to your one or choose 'Inherited'

This option will use default editor colour, so no colouring any more.

Cannot find java.exe in specified jdkhome

If you upgrade your Java SDK, the NetBeans IDE may give you the following error message (Invalid jdkhome specified) at startup:

Cannot find java.exe in specified jdkhome.
Neither C:\Program Files\Java\jdk1.6.0_10\jre\bin\java.exe 
    nor C:\Program Files\Java\jdk1.6.0_10\bin\java.exe exists.
Do you want to try to use default version?

This can be corrected by manually editing the netbeans.conf file found under:

%PROGRAMFILES%\NetBeans 6.5\etc

Replace the line netbeans_jdkhome with the correct path information. Alternatively, you can comment out the line entirely with # and NetBeans will use your default JDK, through JAVA_HOME.

Netbeans - doesn't match expected UUID

In Netbeans using SVN Subversion you get the following error when you try and sync/checkout files.

svn: Repository UUID '1a063a9b-81f0-0310-xxxxx etc' doesn't match expected UUID '7be80a69-xxxxxx etc'
(I've xxxxx'd out some of the UUID numbers in case they were sensitive in nature)

Notes

  • You can't switch from one repository to another. You will have to check out WordPress 3.0 and overwrite your SVN.
  • that you have to backup the custom directories (themes, plugins etc) and then do a new install. So deleting all the existing files/directories and doing a new CO with the WP 3.0 code.
  • The --relocate flag is used to relocate a SVN to a new URL, but it must be the same SVN repo. A SVN has a GUID and the GUID has to be the same at the new location as the GUID at the old location.
  • Files in a local repository have a reference stored in their .svn file, the GUID. These GUID must match those stored in the remote repository. Sometimes they can become mismatched causing this error.

Solution

  • Backup files in your local repository
  • delete all the files including hidden ones from your local repository
  • checkout files again using SVN Subversion.
  • swapped any files that had not been updated and they will then be reflected in the remote repository and a checkout is run again.

My Preferred Netbeans Settings

This will be a collection of preferred settings for netbeans. This will prevent me hunting to change that one setting for that annoying feature.

Disable mysql auto closing

When you press return/enter to split SQL statements, Netbeans automaticlly closes the statement and then re-opens it. This makes the code evern messier.

options --> editor --> code completion --> PHP language --> disable 'Use String Auto-Concatenation after Typed Break'

Netbeans Search

Search is not working

  • make sure there is nothing in the 'File Name Patterns' box
  • mark text can cauyse some issues so go to options->editor->marked occurances, then off/on
  • reboot netbeans
  • make sure you have the correct scope selected

Regex search

These are some examples of searches in netbeans using a Regex pattern

So i want to change some <input> tags.

<input .....>
to
<input ..... />

 The following will search for all <input strings that do not terminate in />

Search String:
<input(.*)([^\/])>

Replacement String:
<input$1$2 />

However this does not take into account of whether there is a space at the end so I need to do another search for:

<input(.*)({space}){2}\/>  - needs finishing

Notes

  • ^ is a 'not' character when within [] 
    [^\/]

Netbeans, Subversion and Windows 7 64-bit…

A minor issue is that the built in Netbeans Subversion integration doesn’t work in a 64-bit install. It helpfully provides a link to the Subversion client software at CollaNet, however the most recent version for Windows is v1.5.5 which isn’t a lot of help if you are using Subversion 1.6.x.

All is not lost though, slik subversion provide a free command line Subversion package in both 32-bit and 64-bit versions. This installed without any problems and all that is required is to set the ‘Path to the SVN executable File:’ in Netbeans to C:\Program Files\SilkSvn\bin .

 netbeans-settings

GitHub

Here I will give the basic instructions on how to use GitHub with Netbeans. I can be confusing at the beginning but does get easier.

  • If you checkout a project first in Netbeans it will ask you if you want to create a project
  • If you split metadata and files it keeps prompting/prefilling with meta location
  • If I use meta data in same folder, what happens 1 folder I can exclude (Netbeans automatically excludes the Netbeans meta folder via the gitexclude.git file)
  • Once a project is checked out the extra functions are available (via menu) for uploading changed etc..
  • If you leave metadata in the same folder as your project it adds a [.gitignore] file and a nbproject folder which I think gets ignored.
  • Keeping the meta data in the project folder is better for allowing you to move and managed projects easier.

Clone rep

  • Clone the repository to the required location
    • (Team/Clone) i.e. to your local webserver root directory
    • Git repository URL is from github and easy to find
  • Select MASTER only
  • Clone
    • You have the option of netbeans checking to see if there is a project for this repository and if not gives you the option to create one.

Create project

Keep files in the project folder

Other

  • A Netbeans tab adds 4x spaces into the code, not a tab character, but interprets 4 spaces as a tab (this is the professional way to do it and maintains code layout between software)
  • In MyITCRM, the smarty stuff is tabbed in using the tab character causing it to look funny
  • Notepad++ can swap tab button behaviour to spaces in preferences/tab settings = Replace by space, leave as 4
  • Professionals say use spaces and not tab character because of formatting issues
Published in Programming
Read more...
Page 1 of 96