Tech: Quirks on Mudcat

Subject: Tech: Quirks on Mudcat
From: katlaughing
Date: 07 May 12 - 05:40 PM

I couldn't find a relevant, recent thread for this, so...Max, I am getting a lot of postings, esp. PMs which don't go through the first or second time. I have to back out of the thread,etc. and start all over trying to post. It's a glitch I've seen over the years, but I don't remember it happening so often as of late.

I am also getting misdirections, i.e., I clicked on an unopened PM and found myself at Camsco Music!

Anyway, figured you might want to know with the new server and all.

Thanks,

kat

Subject: RE: Tech: Quirks on Mudcat
From: GUEST,CJB
Date: 08 May 12 - 05:29 AM

Sounds like a redirect virus. I had one and it took three months to get rid of it. You need to do a Google search for "redirect virus" (and hope that the results aren't directed to a porn site of something). There's a bunch of virus removal apps. that you can download for free. Try Malwarebytes to start off with.

Subject: RE: Tech: Quirks on Mudcat
From: JohnInKansas
Date: 08 May 12 - 06:29 AM

It's also possible that transferring all the files to a new server (and a new databse) has "confused" the drives. While the mudcat database isn't enormous by contemporary standards, the larger hard drives that have become common do seem to have some "lag" in getting all the peripheral data organization in place, and can be temporarily boggled when large changes or relocations of files happen. It does seem that they need occasional "idle time" to get things orderly, although they do a good enough job of keeping it under control if you limit the amount of thrashing you make them do.

I've found multiple ways to befuddle my puny little 1TB C:\ drive to the extent that for some of the strange things I do, when I'm "being strange," I expect "Windows Explorer has stopped working and will be shut down" four or five times per day. For the same tasks I "F5" 60 or 80 times a day because the file management system can't keep up with the changes, so it shows wrong info about what's in the folders. It sometimes takes a few minutes to take it's pill (wherever it gets them) and come back up. That results in "out of service" lags comparable to what I've seen when posting here, although the time-lag is about the only clue I get here.

A few days ago, one post timed out without doing anything. A second try flipped me back to the front page but didn't update the thread I posted to. The third try went right to the verge of a time out, but did post. A half hour later everything was zippy quick and squeaky clean.

Organization is the key, but the file management systems on large drives with lots of recent activity take some time to get their sh*t together when there are lots of recent changes.

At least that's been my theory for my own machine, based on carefully analysed and documented hallucinations I had in a dream a while back.

John

Subject: RE: Tech: Quirks on Mudcat
From: GUEST,999
Date: 08 May 12 - 07:55 AM

John,

Would it have anything to do with Java and Google Analytics, Google AdSense, Facebook Connect, Google Plus 1 and Google Custom Search?

Subject: RE: Tech: Quirks on Mudcat
From: GUEST,999
Date: 08 May 12 - 07:59 AM

Java tracking scripts I meant.

Subject: RE: Tech: Quirks on Mudcat
From: Jack Campin
Date: 08 May 12 - 08:28 AM

As far as the server is concerned, those scripts are just more chunks of data thrown into the page as sent to the user - they only do anything active at the user's end. There's no reason to expect they'd cause any more problems for the database than the message texts.

Surely Mudcat doesn't run on Windows? - there may be problems with large filesystems and databases taking time to optimize on Unix-based servers, but they shouldn't be anywhere near as bad as what JohnInKansas experienced.

Subject: RE: Tech: Quirks on Mudcat
From: GUEST,999
Date: 08 May 12 - 12:26 PM

Jack. I didn't understand what I was asking and you explained it well. Many thanks.

Subject: RE: Tech: Quirks on Mudcat
From: JohnInKansas
Date: 08 May 12 - 05:18 PM

A number of people have reported the same problems I've been seeing, but there's been no comment from any of the big name people. My "working fantasy" is that it's the result of changes in hard drive formats to handle the much larger sizes of files and drives.

On very early hard drives, when a file was added to the root sector the only information recorded was the 8+3 filename, about 4 attributes, a date/time stamp, and the cluster address where the file started on the drive. If there was more than one cluster, each cluster included a "next cluster" address to allow the system to "follow the chain" to retrieve the entire file.

Very soon, an "optional" element to allow each cluster to "point back" to the previous cluster was added, but that information sometimes was added "later" by file maintenance components, and some systems omitted it entirely. The main reason for the addition was that a single bad cluster could prevent recovery of any (or on average half) of a "lost file" and being able to follow the string from either end improved recovery. (We sometimes did a lot of "lost cluster recovery" way back when, although most of it didn't recover much.)

The "fixed size and location" of the root sector permitted simple BIOS bootup, since the BIOS could know exactly where to find the starting point for finding stuff on the drive, but its fixed size limited the number of files that could be tracked there. By replicating the same structure in "folder" files, the root File Allocation Table (FAT) could track its few folders, and the folders could be "expandable" to allow lots of files in any of them.

At about the same time that the "previous cluster" space was added to the standard format, the <EOF> marker was made optional, on grounds that the computer would know it reached the end of the file if there wasn't a "next cluster. Some, but not all, drive managers (the front end built into the drives) always added an EOF since a cluster that was only part filled could be used "up to the EOF" without having to "clean" the rest of the cluster if something else had been written there previously.

Since the computer (or actually the hard drive interface) had to follow a long string of cluster to cluster "passes" in order to find information about what was on the drive, the "daisy-chain" process began to slow down file access and retrieval. A few "new formats" appeared, one or two of which survive.

The most common new format for Windows is now NTFS, in which the "boot sector" FAT is now "movable" and extensible. (At least one somewhat similar format exists that's more commonly used on some servers, but I don't know much about the relative usage frequencies.) Getting rid of the fixed size of the FAT permits a much larger number of "root files" but also allowed addition of a whole lot more information about the files at a "high level" in the logical structure of the drives. The "File Control Block" that now keeps a record of "all the cluster addresses for all the files" was added/extended. In theory, having all the "location" information in one place speeds up file access, and the combined "improvements" allow much better management of large files on large drives.

Unfortunately, in order to keep things speedy, only a bit of "marker data" is written when you add, remove, or move a file. The drive maintenance utilities add the rest of the information, theoretically in "idle time processes" as processor time is available. The additional information maintained for each file now includes a multitude of "attributes" (rumors say up to 128?), long file names (up to 128 or so characters in addition to the 8+3 name that the system creates for it's own use, although Windows has a lower name length limit than the spec says is permissible), search tags (almost unlimited?), and probably some other "secret stuff"** we can only imagine.

** tossed in for our conspiracy theorists.

If you put a lot of stuff onto a drive, and leave it alone there long enough the "maintenance idle time services" will add and organize all the "stuff." No hard information is available that I've found for how long it takes systems like mine to complete all the bookwork, but for a half gigabyte in a few hundred files I see "activity" for at least several minutes. If you copy a drive with a half TB of files, and allow search tags, it can take several DAYS before the drive stops thrashing itself and quits adding stuff to the FCB and to the "folder headers."

If you move or delete recently added/moved files, or make a second fairly large change in the same folder before all the cleanup is done, the system has to "cleanup the cleanup before it can clean up" and it gets "confused." The first symptom to appear is usually that a file added isn't displayed in Windows Explorer (although it usually appears if you hit the F5 "refresh" button). A secondary symptom is that Windows Explorer will say that you're in one folder, but when you "paste" something it goes in another folder that you were in before that. An occasional symptom is that Explorer will refuse to paste a file because it says it's already there, when it isn't. The ultimate failure is "Windows Explorer has stopped responding and will be closed" in which case everything goes blank until Explorer can be restarted. (Win Explorer has to be running even to display desktop icons, and the restart won't reopen the Explorer window(s) you had up.)

Or maybe it's all something else. (Where's Steve Gibson when we need him?)

John

Subject: RE: Tech: Quirks on Mudcat
From: GUEST
Date: 08 May 12 - 06:13 PM

because it is a spinning disk....most "speed freaks " never load more than 50% of a drive....when is the last time you defragged?

Subject: RE: Tech: Quirks on Mudcat
From: JohnInKansas
Date: 08 May 12 - 07:21 PM

Recent Windows versions defrag automatically. Mine runs at 02:00 US Central Time the same day every week. You can manually order a defrag, but if the automatic run has been turned on it won't change anything.

Keeping a fair amount of free space on your hard drive is important, since by default Windows won't generally use more than 20% of the free space for temp files or "virtual RAM." You can change the percent used, but that can cause other problems. (Very old systems were limited to the largest single block of contiguous free drive space, making defrag very important to defrag the free space; but newer ones can span free blocks.)

Unless you deliberately change things, the basic free space requirement only applies to your System Drive, since the computer won't use the free space on other drives for swap files and the like, unless you intentionally tell it to (and you can't even do that with the "Basic" versions of newer Windows that many people run).

You can assign free space on other drives for use as temp space in some programs, but it's only easily done in some of them.

If you rely on manual defrags, defrag itself may require around 20% free space on the drive you're defragging if it's badly splattered, but even that space requirement drops if you run it often enough to keep things orderly. Most people shouldn't need to manually run defrag for Vista or later, although it doesn't hurt to check occasionally to be sure the scheduler is running it for you.

The "speed" referred to above is just fast enough so you don't go to sleep and fall out of your chair. It ain't "performance speed." It's just fast enough for the system to keep up with itself - which in some cases it doesn't quite do.

It's not how much is on the drive. It's how much has been changed recently that bogs things down, since the idle-time (background) updates for one change may not be completed before a sugsequent change occurs.

NTFS tries to keep track of more information than it can handle when faced with multiple sequential changes. Give it time, and it will sort it all out, unless you hit it with another flurry of changes before it digests what it's already working on (which I do about daily at present).

John