Tech: Digital Tradition Programmer Needed

Subject: Tech: Digital Tradition Programmer Needed
From: MMario
Date: 13 May 06 - 08:01 AM

dick greenhaus gave me permission to Post this in a seperate thread - it was his response to someone complaining the Digital Tradition hadn't been updated recently:

If there's a reasonably competent low-level programmer out there who's willing to spend some time--I don't think it should be a great deal of it--working with me, we can come out with a significant upgrade of the Digital Tradition (maybe 1000 new songs) in a reasonably short time.
Please PM me if you have any interest.

PM Dick

iF YOU ARE A NON-MEMBER AND WISH TO HELP WITH THIS PLEASE POST HERE AND/OR EMAIL [e-mail]

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Susan of DT
Date: 13 May 06 - 08:20 AM

or e-mail [e-mail] to get to Dick directly or 800 548-FOLK

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Stilly River Sage
Date: 13 May 06 - 12:18 PM

Would you describe what kind of "programming" skills are needed here? If you're using a friendly interface and it is a matter of plugging the data into the correct lines, that's one thing. If you need to actually know how to program, that's another.

Also you use a notation for the cords and music that I don't know. Is there a trick to it, or is all of this in place and someone simply needs to come along and enter it into the database using a form, etc.?

SRS

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Former member/current occassional drop-in
Date: 13 May 06 - 12:25 PM

It isn't just programming help the DT needs. They need to turn the resource over to people with library development--especially online collection development--expertise. That means an in-depth understanding of cataloguing, cross-referencing, etc.

Which also takes money.

Please, turn the DT & music forum over to someone who knows what they are doing. Please.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: dick greenhaus
Date: 13 May 06 - 12:26 PM

SRS-
No, I'm not talking about data entry; I need some help in programming. Sadly, every time either Microsoft or Apple changes their operating system, the existing programs used in DigiTrad become non-functional. To make it much worse, some folks who were doing the programming for the Windows and Mac versions are too busy with other things to devote the time and energy needed.

If anyone wishes details, please call me at 800/548-FOLK (3655) and I'll be happy to discuss them.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Fibula Mattock
Date: 13 May 06 - 12:40 PM

I have experience of digital archiving and delivering collections online. I can programme. I don't have all that much time, but I'd be happy to advise if I can be of any help. I think it's a fantastic resource (the DT - I can do without the BS threads). It would be well worth putting in for funding via grants, etc.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Former member/current occassional drop-in
Date: 13 May 06 - 01:04 PM

A generous offer, Fibula. I would imagine there is no shortage of young up and comers who would relish an opportunity to turn the DT & music forum into a respectable online collection and archive. They tend to have a lot more time on their hands too.

Sadly, few young programming & collectin savvy musicians come here. I've seen quite a few over the years get quickly run off when they try and do their thing here.

Taking on the DT and music content of the forum is a job tailor made for the bright, dedicated, knowlegeable young people. And they are out there (and not here) in pretty large numbers too.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Cruiser
Date: 13 May 06 - 01:08 PM

GUEST,Former member/current occassional drop-in

How about you, since you brought the subject up? Your contribution would be appreciated by us all.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Amos
Date: 13 May 06 - 01:11 PM

Guest former etcetera:

I dearly wish you would identify yourself when you post.

It is much easier to take criticism seriously when you can associate it with a certain source rather than a bland generality. I would think if you wanted your remarks to be taken seriously you would use some handle, even your "ex"member's handle, and would state your ideas of system design explicitly and specifically. Not doing so leads me to suspect that you don't want to offer serious remarks, and merely wish a safe invisible spot from which to criticize.

When all you do is offer sweeping and anonymous generalizations about how poor things seem to you, you have no more weight than any other nattering nabob of negativity.

A

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Former member/current occassional drop-in
Date: 13 May 06 - 01:28 PM

Amos, I am sure you would prefer it if I named myself, as then you could attack me in the time honored Mudcat way: by deflecting legitimate criticism of how the the DT & music forum resources have been squandered, and focusing your attack on me personally.

One thing you can't quite wrap your inflated ego around though, Amos. I'm not your dancing monkey.

As to why I wouldn't offer to help current efforts to rehabilitate the DT or donate money to the cause?

Max & Dick's track record.

If the DT & music forum archives were to be turned over to someone else, I would closely scrutinize THEIR track record, THEIR funding capabilities to see it through, etc. Just like I do any charitable cause.

The last thing I am willing to do is to throw money into this rat hole, just so the Mudcat junkies can maintain access to their community cesspool.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Stilly River Sage
Date: 13 May 06 - 02:23 PM

This glass may be half-empty right now, but that won't be solved by caustic "guests" or recently-banned-ex-members sniping. Pull your socks up and offer assistance or back off.

What programming language are you using? Is this the Macromedia family of things (Coldfusion) or something else?

SRS

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Jon
Date: 13 May 06 - 02:51 PM

Dick had and been and presumably still is using AskSam for the DT. It's not clear in this thread what he is asking for but I would guess programmers to produce Windows and Mac versions (I think Dick does the DOS version himself).

Driftinng but... I had a play with (the DOS version I think) DT a few moths ago and found I could export it to a plain text file using a free trial download of AskSam. I chose to make a MySQL database from the text and found it quite easy to do.

I must admit to have been toying with the idea of producing yet another version of the DT but rather than using MySQL as I don't think many would really want a client-server database, investigating HSQL or maybe SQLite as the database and using Java.

It's way down on my list of things I want to though - I might never get round to it - but it may be an idea someone would like to try - a cross platform version.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Mick Pearce (MCP)
Date: 13 May 06 - 05:00 PM

Jon - I did something similar when I was working on the abc version - I exported the DT into an Access database with tables for the song info, texts, tunefiles and keywords. I still have that if anyone wants it (about 33Mb).

Mick

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Jon
Date: 13 May 06 - 05:21 PM

Ditto to the MySQL version if anyone should want the current dt in that format, either as tables or an SQL dump. The tunes in this version are just in the SongWrite format.

Mick, if you are willing, I would like a look at the abc conversions some time.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Mick Pearce (MCP)
Date: 13 May 06 - 05:52 PM

Jon

You're welcome to them. I have 20-odd zip files containing more or less the song files starting with each letter of the alphabet (it's the way John in Brisbane wanted them). I'll give a sample below so you know roughly what's in each file. If you want them, email me at MickCPearceATaolDOTcom with an address to send them to. (Ideally it would be good to get hold of the versions John in Brisbane was working on before his untimely death. He'd done some work cleaning up the files into a better final version).

Mick

---

Sample file: AKDRUM.abc

X:1
% File Generated from SongWright AIKDRUM by Mick Pearce
T:Aiken Drum
M:2/4
L:1/32
Q:1/4=100
K:C
V:1
K:treble
z8G8|E6E2E4C4|F6F2F4C4|E6E2E4C4|D6D2D4F4|E2E6E4C4|F6F2F4F2F2|E4E4D4D4|
w:There was a man lived in the moon, lived in the moon, lived in the moon, There was a man lived in the moon, and his name was Ai- ken
%W:chorus:
C8G4G4|E4E4E4C4|F8F4C4|E8E4C4|D8D4F4|E6E2E4C4|F8F4F2F2|E4E4D4D4|
w:Drum. And he played up- on a la - dle, a la- dle, a la - dle, he played up- on a la- dle And his name was Ai- ken
C8z8|
w:Drum
W:
W:Words source file: AIKDRUM
W:AIKEN DRUM
W:
W:There was a man lived in the moon
W:lived in the moon, lived in the moon
W:There was a man lived in the moon
W:and his name was Aiken Drum
W:
W:And he played upon a ladle, a ladle, a ladle
W:He played upon a ladle and his name was Aiken Drum
W:
W:And his head was made of cream cheese...
W:And his name was Aiken Drum
W:
W:And his coat was made of good roast beef....
W:
W:And his breeches were made of haggis bags....
W:
W:(continue to ad lib)
W:
W:@food @kids
W:recorded by David Sear
W:filename[ AIKDRUM
W:play.exe AIKDRUM
W:SOF

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Jon
Date: 13 May 06 - 06:34 PM

Thanks Mick, email sent.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Bill D
Date: 13 May 06 - 06:36 PM

Former member: as Dick G. has said many times, the DT database is 'copylefted'. It can be, (and has been), copied, reformatted, added to, and otherwise tinkered with. This collection was begun 14-15 years ago, a labor of love.

If they had waited for 'experts'...whatever they were back then...to be available to 'do it right'...or your notion of right, it wouldn't be here at all. It is what it is, and it has been invaluble to many for over 10 years....even before Mudcat.

Anything of this nature can be improved, especially by throwing money at it, but your criticisms go way beyond 'constructive'. You are demanding both some high level of scholarship AND some professional level of programming to be applied to 10,000 songs....and once done, you'd no doubt find fault with the font used!

This is a list of song lyrics with lots of tunes, hosted on a friendly site and tied to an older database system that is, obviously, tricky to update.

As you see, attempts ARE being made to improve things! Now either help, or go away.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Big Mick
Date: 13 May 06 - 06:40 PM

S/he won't go away, Bill, as s/he has this superiority complex. It is borne of a life of talking about problems from a velvet seat. Same old stuff. S/he will now hurl filth, epithets, and aspersions my way. But s/he will not do anything to support the work.

Jon and Mick, let me know if there is anything I can do to help in the process. I am not terribly good but I can be taught and have some time to put in some keystrokes.

All the best,

Mick

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Bill D
Date: 13 May 06 - 06:43 PM

Yeah, well...I don't doubt you are right, Mick...but I wanted to see in print what others might need to know about the situation.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Big Mick
Date: 13 May 06 - 06:45 PM

JON, forgive my being out of touch, but is The Annexe no more? I had wanted to drop you a personal note, but found it not there.

Back to the DT discussion.

Mick

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Jon
Date: 13 May 06 - 07:15 PM

Two bloody Micks!...

Big Mick, I think we are talking cross purposes here. MCP and I were talking about work already done for whatever reasons we did them.

My play came from some conversation in the help forum with Joe Offer. the results can at least for the moment at least be seen here

I wanted to see MCP's abcs part as curiosity as to how they came out and part as with them, I can make myself a nice local copy of the dt, adding some more modified modified folkinfo code and I could print a pdf with sheet music or produce a midi of the tunes in any key from hia abcs.

Anyone can have a copy of the existing php code if they want it btw. A version with the abcs. I don't know how MCP would feel about the abcs being included in such a database for anyone that wanted but in any case, I don't really see it being much use to many as without being funny, it does involve rather more setting up than a typical Windows install.

I don't know if I will get round to looking at the cross platform version I mentioned trying but if I do, it will be a long way off.

--------
As for a more general comment re the dt. I still think the ideal would be for Dick and Max to come to some sort of arrangement. I presume the online copy here is stored in a std SQL database. If Dick could scrap AskSam (and preferably move to abc from Songwrite) and Max provide a means of him inputting directly to the MC dt database I think life would be much easier.

The online MC dtdatabase need never be badly out of date that way and it should be easy to provide say a 6 monthly snapshot of the database, which should be in a formmat which I suspect most potential programmers would prefer to AskSam and perhaps someone taking a copy that way would look to just using/importing/exporting the database tables, updating just them rather than what I get the impression is more of at least a partial reprogram for every one of Dick's updates.

Oh well...

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Big Mick
Date: 13 May 06 - 07:31 PM

OK, Jon. But what about the question about The Annexe? How do I get in touch or log in?

Mick

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Former guest
Date: 13 May 06 - 07:50 PM

I was also talking philosophically about getting DT/Mudcat out of the proprietary "I control" mode of "updating" (which has really only ever meant making additions and a few edits) that Dick and Max have, and moving towards a wiki "we collaborate" mode where many people could be working on it at once.

But then, DT/Mudcat is a control freak, hierarchy driven sort of place. Rather the antithesis of the wiki collaborative (and much more democratic) model for grassroots endeavors like the DT/Mudcat music archive.

The whole "well, it's Max's site" mentality around here says it all.

The beef so many people have with Max & Dick show is it ain't THEIRS. The collection should belong to the online music community, and be controlled by the OpenEdit spirit of our online music community, as happens in wiki communities. The online folk & blues community is HUGE. It isn't just this wee backwater.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST
Date: 13 May 06 - 07:51 PM

Sorry I missed that one. Mick,

The Annexe is still there at http://www.jonbanjo.com/forum and also there is now another copy at http://www.jonbanjo.org/forum which can be tried if jonbanjo.com goes down.

You might have been looking for a specific page asp page and maybe I did something wrong in my attempts to redirect such requests to a new php page.

What happened was my ISP did an overnight runner on me leaving me with all sorts of problems. One of the things it prompted me to do was to rethink the way I was operating and to try to at least provide myself with better cover against such events. Part of that move was to rewrite everything in php, enabling me to host from home (although the Annexe is currently on shared hosting) using entirely free open source software. I know it did cause some confusion, but I think we do have better protection against such happenings now.

-------

Anyway, as you say, back to the dt...

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Mick Pearce (MCP)
Date: 13 May 06 - 07:53 PM

Jon

I've no problem with the abcs being used in any way.

Mick

Subject: RE: Tech: Digital Tradition Programmer Needed
From: The Fooles Troupe
Date: 13 May 06 - 08:00 PM

The problem with 'legacy' systems is that they always need 'crutches' every time the world changes, especially the Microsoft World changes, as it will do in about 12 months time.

The problem with 'upgrading' legacy systems to the latest Microsoft system is just the same - the whole damn job has to be redone every time MS sneezes.

Cross-platforming has worked wonders in the 'Linux' type system development world - and most of the tools are free - most of the more important ones can even be cross ported back to Windows.

I'm with most of the last 2 paras of JiKs comments above. Also, in spite of people's attachment to various other proprietary music formats, midi and ABC are gaining ground as the the most widespread systems for the type of simple music (tunes) needed here. Of course, one would have to check that any move totally into ABC used the most widely compatible 'version' of ABC standards.

Btw, I don't really see the need for wavs or mp3s - unless you wish to preserve EXACT performance details of various artists. We never had that with written records any way.

Robin

Subject: RE: Tech: Digital Tradition Programmer Needed
From: dick greenhaus
Date: 14 May 06 - 12:57 PM

There seems to be a good bit of confusion, and a great deal of misinformation, about the Digital Tradition. Maybe this will help clarify matters.

DigiTrad was and is intended to be a songbook. While I insert as much background material as I can, it can never meet the standards of a scholarly study-nor should it. It contains errors and misattributions-a necessary consequence of soliciting and including contributions from anyone; these are corrected-sooner or later-as time permits and newer editions emerge.

Quite early on, I decided that the only acceptable structure for this sort of collection is a full-text searchable database. Titles are virtually useless for identifying what you're looking for , as many songs share the same title (I can think of half-a-dozen or so called "Lady Margaret") and many songs are referred to by more than one title. A decent full-text search, expecially one that permits the use of "wild cards" as well as Boolean operators such as "NOT", "OR" and "AND", and one that can search out complete phrases as well as individual words is, IMO, the only way to go. Linking the lyric with a playable and printable melody is another extremely important feature. And displaying the words in a display synchronous with the tune is, I believe, another basic requirement. We were able to accomplish all of these with two programs-AskSam and SongWright.

A decade or so ago, we had all this. Then some unforeseen things happened. Firstly, we found that the number of hits we were encountering on Mudcat made it imperative to reduce processing time (for a search) to a minimum; while storage space didn't seem to be a problem, searching time was. As a consequence, the version of DigiTrad accessible online is in several ways a pale shadow of the DigiTrad you could get on disc. The full-text search was weakened drastically, and the playing of the tunes was in MIDI format, and could not be synchronized with the lyrics.

At about the same time, a couple of things happened to the disc-based system: The DOS operating system became demonized and, to a great extent, made obsolete by the advent of Windows. An increased demand for a Digital Tradition that would run on a Mac emerged. And the scheme I used for playing tunes-depending upon programming the computer's "beeper"--became non-functional when the newer computers came out without beepers.

A brave volunteer programmer came to our rescue-at least in part-by converting the DOS program to a Mac program, and incidentally, to one that worked on Windows. While using a search engine considerably less powerful than the original, these new versions had the not inconsiderable advantage of displaying a printed score, as well as playing the tune in a MIDI format while changing the color of the notes as they were played.

All was good--not perfect, but good. And then both Microsoft and Apple came out with upgraded operating systems; not just once, but several times. And each time the operating system changed, we had to modify DigiTrad to not only accept these changes, but to remain compatible with all the previous versions.
To make things worse, our volunteer became involved with a major project in the folk world having nothing to do with The Digital Tradition nor Mudcat., and has been unavailable to us for several years.

OK. Where does that leave us? Susan and I have over a thousand songs ready to add to the Digital Tradition. Neither of us has the low-level programming experience to prepare the next release. I don't believe that the programming should be too difficult: The lyrics (presently in AskSam format) can readily be exported as text files; the tunes, (in SongWright format) can fairly readily be converted to MIDI or ABC (there are programs in existence that do this, and I believe we can obtain a license to use them.)

The next edition---really four editions-must come out in HTML format (for Mudcat), Windows format (to handle XP and previous versions) and Mac (to handle whatever's the latest version and back through the various OSs.)

Some last points. Neither Susan nor I are funded in any way for the work we do on DigiTrad. And, in response to a nameless Guest who has stated:

"If there were some way to wrench control of the DT away from Max and Dick"
"It is my understanding (and someone will correct me if I'm wrong, which is fine) that there was a marriage of Max to the DT because of Max's expertise in data mining."
"Many people active in the folk music community in North America, Europe and beyond who either never posted to the forum or no longer post here, happen to recognize that DT and the music archive will only have long term value to the music community if control of it is wrested away from Max & Dick."

I'd like to point out that Max's involvement with the DT consists solely of providing it with a home. As far as "wresting" or "wrenching" control of The Digital Tradition, I can only ask, "What control?" Anyone is free to use DigiTrad as they wish, and several people have.

And I'm still looking for programming help. Call 800/548-FOLK (3655)

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST
Date: 14 May 06 - 01:57 PM

Yeah, we saw how well Jon was treated when he wanted to help.

The DT & Mudcat music archive need to be wikified so ANYONE can contribute, clean it up, edit, etc. Not just those you hand pick.

There are many people out in the online music community with much more expertise than the Dick & Max Show, and would leap at the chance to make it into something useful and useable.

Think open source. Think Linux. Think out of the [bleep][deleted for gratuitous vulgarity] box.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Stilly River Sage
Date: 14 May 06 - 02:02 PM

Thanks for the explanation of the process, Dick.

I wonder if a summer internship for an upper-level or graduate-level university student in computer programming might be an answer? There are many jobs that need to be done that a talented student can undertake in a summer's time and bring new blood to the process. It would require training, supervision, and perhaps a stipend of some sort. But treat them well enough and maybe they'll come back to help again. :)

SRS

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST
Date: 14 May 06 - 02:23 PM

Or just wikify.

Why is it so important to people here that the resource be squandered here, instead of made great by those capable of doing so?

Answer: the truth would be put to the lie of "Mudcat is such a wonderful music resource and a great community".

You all live in fear of being just the forum, despite the fact that is all this place is about anymore. The clubhouse forum membership.

You drive off guests and newbies alike, to keep it clubby for the membership.

Now people are talking about "the fix" to resurrect Mudcat to it's former glory being making it an OFFICIAL online membership only forum.

Yeah, that will fix all us bastards who come here for the music.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST
Date: 14 May 06 - 02:35 PM

Try reading what people write rather than making up your own arguments to put into the mouths of people you wish to make personal attacks upon.
Dick has just said that ANYONE can do WHATEVER THEY WANT with the information - tunes and words contained within the Digitrad. Many people have already done so.
If you want a Wicki - set one up. Dick has already said that the Digitrad contents are freely available.
Quack!
GtD.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Big Mick
Date: 14 May 06 - 02:44 PM

Geoff, this isn't about facts. It is the same old ogre up north that loves to hear herself talk and congratulate herself on how brilliant s/he is.

The DT is what it is. Dick and Susan have made it freely available to do what you will with it. Some folks need to quit listening to themselves talk and put their money where their mouth is. But of course that would require real work instead of pontificating.... and as we have seen in many threads, that is beyond the ken of this person.

Dick, the idea of searching out a grant and an intern sounds like it might be viable. Waddya think?

Mick

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Stilly River Sage
Date: 14 May 06 - 02:50 PM

Trouble with the Wikipedia model is that even more mistakes than usual end up in there. You either put the information in the way it needs to go, or you spend your time running around fixing mistakes entered by people who don't have the entire story, who haven't read the instructions, or who have an ax to grind and are doing it their own way despite the accepted practice. This doesn't mean that it will all go in wrong, but it does mean that there are quite a few sour folks and blow hards who would put the credibility of such a project into question. Just as exists at Wikipedia today. I don't use it as a source. When one of my searches hits a Wiki page I will take a look at it, but I always go looking for independent credible verification.

SRs

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Jon
Date: 14 May 06 - 03:14 PM

Dick has just said that ANYONE can do WHATEVER THEY WANT with the information - tunes and words contained within the Digitrad.

Quite true Geoffbut the biggest problem would be DIYers like myself have is that so much gets stuck in Mudcat threads for so long. And, no, I don't think it a good idea to have several "custom" dts floating around with differnt selections of harvesting, filenames for tunes, etc.

BTW, I get the impression from Dick's post that he hasn't understood my comments.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Geoff the Duck
Date: 14 May 06 - 03:55 PM

Hi Folks - found my cookie - it was under the tea cosy!
Mick and Jon - yes I know that the troll is not interested in the Digital Tradition and is just using it as an excuse to be obnoxious. I just thought that somebody ought to point out for the benefit of any legitimate visitors that the Digitrad is NOT a private empire.
I also do not see the Wiki (arent they just short Wookies anyway?) idea as a sensible approach, for the same reasons quoted by SRS and yourself, Jon, plus the fact that it is NOT JUST a collection of lyrics. Dick's aim of having words and melody tied together accurately needs a well coordinated system. BUT - if somebody did want to produce a Wiki, there are no restrictions imposed by Dick which would prevent such an endeavour. Any insinuations are simply deliberate obfuscation (nice word), not the reality of Dick's position.
Dick - I do not have the correct technical background to be of help computer-wise, (give me a field of fungi and I could catalogue THEM for you), but would support the effort if I had.
Could you try a University Computing department - perhaps they could be persuaded to let a student use the programming challenge as a coursework project?
Quack!
GtD.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Desert Dancer
Date: 14 May 06 - 03:57 PM

I've been trying to filter constructive suggestions out of Guest's vitriol.

Jon said, "I don't think it a good idea to have several "custom" dts floating around with differnt selections of harvesting, filenames for tunes, etc."

Why not? Assuming appropriate acknowledgments are made for the initial dataset, and a clear statement that the wiki (or whatever) was a separate beast (I think this would be important), if Guest truly has a larger army of folkies than visit Mudcat, why not? Why limit the resources available in the world? Presumably those who cared to could harvest from the MC Forum as well as any other source, as we all do (I would hope that such a Wiki would include in its groundrules proper citing of sources), certainly the reverse would be true.

Would it confuse those searching for songs? In this Google information age, we're not limited to any one resource; I doubt that many of us Mudcat devotees consider the DT the definitive source of anything.

Would it dilute the effort of compiling a great collection of songs? Maybe, maybe not. There lots of other collections and collectors out there now. I don't know if I want to say "the more the merrier," but, I find it hard to say it's a bad idea for someone to make a different run at it.

A more technical question: Would it be possible to maintain a downloadable database via a wiki interface? Or would a user be limited to working online?

I don't mean to distract from Dick's search for a programmer to maintain the current vision of the DT. Maybe this discussion should move to a separate thread.

~ Becky in San Diego, at the moment

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Geoff the Duck
Date: 14 May 06 - 04:11 PM

Let's face it, one problem with the Digital Tradition was that such a large collection of lyrics by necessity involved contributions from many people, some of whom provided sets of words which were misheard, incorrectly ordered or otherwise mistaken. The process of simply correcting such errors is I am sure, a mammoth task in itself. Setting a large number of people onto a task is not necessarily an effective answer. Small focussed teams usually do a better job (too many cooks DO spoil the broth).
Correcting ONE central source as part of the ongoing process of updating in general makes much more sense. Keep up the good work, Dick and Susan.
Quack!
GtD.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Jon
Date: 14 May 06 - 04:17 PM

DD, I for one would probably want to get back in sync with a new official release for one reason.

As for WIKIs, I'm afraid I haven't got round to looking at them fully enough yet. I have toyed with the idea of turning folkinfo into or including a WIKI at folkinfo. In some ways my gut feeling is that if I could encourage the use and get contributers, it might suit the nature of the site, perhaps with articles on the collectors, backgrounds to songs, etc. At first glance, it does seem to offer advantages over having to search for stuff in threads. Maybe this year I'll get round to doing some research...

As for songs in a WIKI, I don't know. One area of doubt would be the structure and ease of searching using facilities such as the full text searches. As far as I can gather, trust doen't usualy seem to be a major issue and on accuracy, with co-operation, it could have the reverse effect to the one SRS suggested. Trouble is, as far as can see, you wouldn't knnw till you tried it...

Would it be possible to maintain a downloadable database via a wiki interface? Or would a user be limited to working online?

I'm not really understanding you. What are you thinking of doing?

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Jon
Date: 14 May 06 - 04:23 PM

Correcting ONE central source as part of the ongoing process of updating in general makes much more sense.

I'd go along with one central source, Geoff, but I would see that one central source as being one that could be updated say by a trusted team of say 5-10 people as an advantage.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Geoff the Duck
Date: 14 May 06 - 04:31 PM

Agreed, Jon. It would also be a good thing to have goodwill between different sources (such as Folkinfo) allowing them to feed corrected sets of lyrics back into the Digitrad.
Quack!
Geoff.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Jon
Date: 14 May 06 - 04:42 PM

It would also be a good thing to have goodwill between different sources (such as Folkinfo) allowing them to feed corrected sets of lyrics back into the Digitrad

Interesting idea, Geoff. I can't see how to apply it though. There is goodwill from folkinfo in the sense that links are sometimes given to relavent discussions and songs here and visa versa.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Stilly River Sage
Date: 14 May 06 - 04:47 PM

The Wiki software and/or program design may live only on the server side of things and may not be something that can be saved to a CD or DVD.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST
Date: 14 May 06 - 04:48 PM

It doesn't have to be at Wikipedia, it could be any sort of Creative Commons licensed, open source RDBMS (relational database management system) site where anyone with some basic expertise could add, edit, etc. to maintain a CURRENT AND ACCURATE database and archive. You could download Art Thieme or Jean Ritchie or Frank Hamilton performing the DT tunes, if they are willing to do the performing, and want you to have access to it.

There is just so much MORE that this music resource could be.

Such a site would allow us to draw on the expertise of the many, rather than have what little is here being guarded by a select few.

It is, after all, the people's music.

With such an open source, anyone can access and update the database (for those World Series/World Cup topical songs!), folks with bug fixing expertise come in and fix your bugs, freeing the musicians with content expertise to do the data mining, relational database management, etc.

There are many different software systems and approaches to consider, and that ain't my area of expertise. It also isn't Max & Dick's apparently.

But it should be if the online folk and blues music community is serious about turning the DT/Mudcat archive into something that reasonably resembles an online music archive in the 21st century.

Look, I'm a luddite when it comes to this stuff. But if I know at least this small amount about the POSSIBILITES, w/open source licensing, databases, podcasting, etc, why don't we hear anyone around here talking about making DT & the Mudcat archive great, rather than cumbersome, outdated, and mediocre?

Why aren't all of you demanding the DT & Mudcat archive be made BETTER AND MORE ACCESSIBLE TO ALL???

Why settle for this bogus forum politics crap, when the sky is the limit?

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST
Date: 14 May 06 - 04:54 PM

SRS, I don't think it's quite that cut and dried.

As an example, I could sent you a blank copy my folkinfo database, ask you to add lets say 50 songs, save it to CD and send it back to me and I could merge the 2 sources.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Desert Dancer
Date: 14 May 06 - 05:05 PM

"There are many different software systems and approaches to consider, and that ain't my area of expertise." Aye, and there's the rub. I'm just starting to learn about that stuff myself, and am in no state to actually volunteer any technical skills. We are all casting about here...

I have never downloaded the database myself, but have just this morning taken a look at Amos's copy on his Mac. It's pretty cool. Whether it's more of the same, or something new, it does want a pretty strong programmer or programming team with time on their hands... Of course, there often is a pretty strong correlation between folkmusicalness & geekiness. If we haven't the expertise here at the 'Cat yet, where might we go fishing for such?

~ Becky in SD

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST
Date: 14 May 06 - 05:06 PM

You can download and burn to CD damn near anything on the internet. That is OLD technology.

Have a look at a folk tune wiki here.

If you go to the bottom of the page & click on About Folktunes it shows the tech support used to create their new site.

If you then click in the box at the bottom on the "expanding it" link it takes you to the log-in page, where you log-in to edit.

Not so complicated, now is it? You don't need programming knowledge to contribute, or at least not much.

But there is something you need that is in far too short of supply here at Mudcat: trust of strangers, of the unknown, of the competence and integrity of those who wish to contribute.

Don't get me wrong, it won't end the flame wars (think in terms of a "what is folk" discussion that anyone can edit/contribute to, and no one can censor or delete). But it would be a far cry better than letting the DT & archive languish here, buried beneath a bullshit & meet up forum.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Jim Dixon
Date: 14 May 06 - 05:10 PM

Couldn't the DT be published as one big text file?

Then people could organize it, format it, index it, search it, etc. with whatever software they like.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST
Date: 14 May 06 - 05:23 PM

Here is Wikipedia's Folk music page

Substandard in SRS' and the Mudcat Forum's world, I'm sure. Just not as good as what is here at Mudcat, which everyone here knows is the world wide web gold standard for folk music.

And here, for those of you who aren't aware of what others have attempted to do, is a sample of what just one person can do.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: dick greenhaus
Date: 14 May 06 - 05:34 PM

Jon-
I don't think I missed any of your points. There's no real advantage (and several disadvantages) to my shifting from AskSam---any of the information is easily exportable in ASCII format, and if I dropped AskSam I'd be leaving a fast, very flexible full-text search engine for...??
As far as Songwright is concerned, I'm using it simply because I find it convenient. Once again, exporting to MIDI or converting to ABC is not a serious problem.
In any case, unless someone comes up with a single database engine that works equally well on Mac OS 6, 7, 8, 9, 10 etc; and on Windows 95, 98, 2000, ME, XP,XP Pro etc; and on Linux platforms that also meets Max's requiremet of minimum processing time to (pause to take a breath), we'll still be dealing with multiple versions, so it makes no difference what I use.

My personal belief is that many, if not most, of the errors and misattributions in the DT are due to the large number of contributors. Wickifying can only exacerbate the situation. If someone wishes to create such a system, of course, they should feel free to do so...DigiTrad if free for anyone to use as he/she wishes.
    Jim Dixon--The DT, at least in its DOS format, is readily convertible to a text file. Don't know what you'd do with it once you got it, though.( You'd lose all the tunes, for one thing.)

A last (for now) thought. Until the advent of all the new operating systems, and the need for coming out with all the different versions, the DT was upgraded at least every year. Some programming help--primarily aimed at making the generation of all those versions an automatic chore--would permit more frequent upgrades, and much more rapid response to those who point out errors, misattributions or who have additions.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: JohnInKansas
Date: 14 May 06 - 05:48 PM

GUEST 14 May 06 - 05:06 PM

I looked at your link. It's very pretty. What is it?

Seriously, I couldn't easily find that there is any content. Several links that I tried led to mostly blank pages with nothing I could identify as content. I found nothing telling me what it is, what its intended use is, or any reason I should work harder at figuring it out.

The colors were nice though.

I'm afraid the site is much like a lot of GUEST posts we see. There appears to be an "idea" but it's not expressed in terms that the rest of us can quite get a grasp on. One suspects an encounter with a person experienced in writing Microsoft tech manuals, since a lot is said, but there's no meaning or content visble.

John

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST
Date: 14 May 06 - 05:58 PM

Sorry John, I don't have a cure for stupidity.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: The Fooles Troupe
Date: 14 May 06 - 06:07 PM

"Setting a large number of people onto a task is not necessarily an effective answer."

'It still takes a minimum of 9 months to produce a baby, no matter how many women you set working on the task.'

- The Mythical Man Month

Subject: RE: Tech: Digital Tradition Programmer Needed
From: hesperis
Date: 14 May 06 - 06:08 PM

What would probably work well would be a semi-open wiki, rather than totally open one (or a rather closed one as it is now). Each entry could have comments, the team of moderators would check comments for accuracy and update the entry itself when new facts come to light. People would not be able to edit the original entry itself without being volunteer staff.

Meanwhile, instead of waiting on the moderators, people browsing the collection would be able to evaluate the comments using their own judgement.

The "common people" could just send in a new song or version with a form, then moderators would check and click "approve", rather than waiting on hard-coding the data.

The difficulty in coding such a system would be the cross-referencing, the tunes, a really robust search, and setting up permissions on who can do what to the information.

I'd also add in a song rating system, so instead of 300 comments along the lines of "Ooh, I love this song!" people can vote that they love it, and keep the comments for facts about the song and it's background history.

I've been doing more programmatical (is that a word?) websites lately in php/mysql, and do have a lot of time on my hands. However to take a huge project like this on I'd definitely need some pay.

Perhaps the mudcat complainers would like to get a fund together for hiring someone to make a semi-open wiki version of the DT?

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST
Date: 14 May 06 - 06:08 PM

Imagine my surprise that Dick doesn't trust people enough to allow them to add content to the DT.

Or that others here don't trust open source models of collaborative knowledge generation.

You live in a highly controlled, isolated, insular little world here in Mudcat.

That you don't trust what others are doing out there in the brave new web world just proves that.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Jon
Date: 14 May 06 - 06:15 PM

There's no real advantage (and several disadvantages) to my shifting from AskSam---any of the information is easily exportable in ASCII format, and if I dropped AskSam I'd be leaving a fast, very flexible full-text search engine for...??

By my suggestions for the input source and method which could have course be impossible but I said waht I thought would be the ideal.

An industry standard SQL database
A system that could reflect corrections and additions viewable by Mudcat as and when made.
A system that could again easitly be exported to ASCII amongst other things.
A system that would allow you a team of contributers if you desired.
A syastem that should be able to perform all of your search requirements.
Freedom from some degree of tie to proprietary software.

Have a play with my play here sure it wont do the boolean searches (except AND is implied on a keyword seach and OR is allowed on keywords (without the @)) in some way.

Bear in mind that was little no more than a single afternoon's hack wich had a further couple of 10 minute hacks added as a result of comments from Joe. It's only a play but I really don't think it would be too difficult for anyone to provide all the features you suggest on with free open source industry standard tools.

At least that's the way I see it. Thanks for the discussion btw.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: The Fooles Troupe
Date: 14 May 06 - 06:18 PM

"You can download and burn to CD damn near anything on the internet. That is OLD technology."
&
"Look, I'm a luddite when it comes to this stuff. But if I know at least this small amount about the POSSIBILITIES,"

'Possible' is not always achievable.

A wiki is a large collection of pages, pushed out by a server. How the hell would you load all these cross-indexed pages on to a CD? You would need to have a copy of EVERY page - at a particular instant in time, AND the full wiki server on the user's PC (or LAN).

It MIGHT be possible to grab 'snapshots' and dump them to CD, but that takes some regular stuffing around, LOTS of processing time, and once it's on the CD, HOW do you access it - well you need the wiki server....


"To the ignorant, all things are possible"

"You can't get there from here"
- old Irish anecdote

Subject: RE: Tech: Digital Tradition Programmer Needed
From: DMcG
Date: 14 May 06 - 06:27 PM

I'm not sure how GUEST 14 May 06 - 05:06 PM related to other guests, but that post contains some points that are worth discussing in detail, and I think have relevance to other comments in this thread. For what it is worth, I am been involved in developing computer systems for more years that I care to recall and my current and last jobs have both involved developing some fairly hairy multi-platform, distributed web applications. That is not to claim my thoughts have any special merit; more that I've made a goodly share of wrong decisions in my time.

So let's take a few of the points. You can download and burn to CD damn near anything on the internet.. Yes, you certainly can burn a single item. Burning inter-related items so that the relations are maintained is much more difficult and not a task for the faint-hearted. Sure, there are tools that can help, but don't just assume they can sort it all out. A trivial example: on Unix platforms, the case of filenames is significant, on Windows it is not. Unless you have played that game, you would be surprised how many times that is at the root of real-world problems. Similarly, the set of permitted characters in filenames differs - the most obvious being the forward-slash, backward-slash issue, but there are plenty more. And another problem is deciding what exactly gets 'burnt'. Do you burn everything that is within 1 link of the page you are dealing with? Two? Have some interactive dialogue saying I want this subtree not not that? It's much more tricky than it sounds.

Then there is the question of how the 'next-generation' DT is to be used. For historical reasons, the DT evolved as a stand-alone system. It was, and is, fully usable on a laptop away from any network or wireless hotspots. To build a Wiki-like system where everything can link to MP3s all over the net is great, but it comes with the need for either a good network link - and don't forget there are still a lot of 56Kbit modem users around - or with a very well considered way in which it can 'degrade' when not connected to the net. I'm not saying one requirement is superior to the other, necessarily, but you need to weigh up the pros and cons before you leap into it.

Then there is the software used: If you go to the bottom of the page & click on About Folktunes it shows the tech support used to create their new site. A lot of the problems with the DT come from the decision to use AskSAM. That was probably the best decision at the time, but over the years has come back to roost. How do you ensure that whatever decisions you make about the technology minimise these issues in the future? There's much more to it than simply saying use open source. I've spend about 15 minutes browsing the example site and it makes use of at least five different emerging standards. In all probability, at least one of these will fail by the wayside in a year or two. What happens then? It's one thing to say, as Jon did, that he was considering Java and HSQL: that's a closed and controllable set of constraints. It's another when you start throwing all sorts of things like WebJay into the equation. Nothing wrong with WebJay as such, but is that going to be the dominant solution in a few years time? Do you remember Veronica and Gopher? Or how Netscape was going to make push technology a dominant part of the web experience?

Next contributions: Not so complicated, now is it? You don't need programming knowledge to contribute, or at least not much.. This is the greatest strength of a web-based approach. ANY system that needs a manual intervention to get from a contributer providing a piece of information to letting others see it will inevitably lead to delays because it takes time and effort for the people concerned to move things across. That's the real world for the DT as it currently exists. And its also the same for moderated sites like some the BBC message boards, where they have people employed full time to keep the transfers going, but want to monitor each post for legal reasons.

Trust of strangers, ... , of the competence and integrity of those who wish to contribute... (I left out 'of the unknown' because I think that's a different issue) This again goes back to what the DT actually IS. It has never been a reliable source of lyrics and tunes or history in the sense that an academic would mean that term. So I don't think there is too great a risk in giving a more immediate facility to update the database, were this possible.

But the biggest danger, in a way, comes from the idea of multiple copies. As the DT stands, we have essentially major releases, the last being Spring 2002. That's an age, I know - as do Susan and Dick. But if you have lots of copies all being maintained and updated separately you will not necessarily have something better; what you will have is a number of disparate sources that grow ever further apart. Again, you may feel that the benefit outways the cost. All I say is think carefully before you choose.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: The Fooles Troupe
Date: 14 May 06 - 06:41 PM

"A trivial example:"

I just LOVE your subtle sense of humour DMcG... ;-)

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Jon
Date: 14 May 06 - 10:04 PM

One of these times I will get that link right in this thread...

http://www.fokinfo.org/dtdatabase

Subject: RE: Tech: Digital Tradition Programmer Needed
From: *#1 PEASANT*
Date: 14 May 06 - 10:14 PM

I am a folklorist not a tech person...but have been doing web pages for a long time...

I like the large text file idea. One would put the keywords at the top rather than at the bottom on the same line as the title if possible. That is a song with the word love in the lyric would come up but also a love song would have the keyword love added to the title line. That way when you search for love song it would come up at the top of each instance.

Just open in your web browser and search.

The problem would be integrating the midi play files as they are now.
Not as advanced in html as I should be....but could it be possible to have a search cover both a central frame and border frames.

That is put the abcs in one file in a frame to the left put midi file links in one frame to the right and put the huge text file of all songs in the frame in the center.

If you could get one search button to search each file- left right and center the song would come up like a one armed bandit midi abc and lyrics and information all at once one after another.

These days computers can download much larger files quicker so the huge text file would be big but probably not prohibitive any more.

Just a thought.....probably ill informed because it is not all that informed at all but a thought none the less....

Conrad

Subject: RE: Tech: Digital Tradition Programmer Needed
From: dick greenhaus
Date: 15 May 06 - 12:54 PM

Jon-
THis looks quite plausible. A couple of questions:
a)is it a difficult task to link each tune to a program (Noteworthy, perhaps) that both plays the tunes and displays the notes?
b)How big a job is it to upgrade (additions and/or corrections)?
c) Is this strictly Web based, or can it exist as a stand-alone version? If the latter, what platforms will it run on?
d) How big a job was it to do the conversion (is it sufficiently automatable to permit frequent upgrades?)

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Jon
Date: 15 May 06 - 02:11 PM

Dick,

Firstly I'd like to take a step back...

I'm floating ideas around. The one you have looked at is part of a "system idea" that you could perhaps adopt as both a revamped "Mudcat dt" and your imput source for yourself (and whoever you wished). I was really demonstating the idea and obviously any further discussion, should you wish to go the "whole hog" as it were would have to involve discussion between Max and yourself.

I'll attemt to answer your questions.

a) No,, I don't think that should be a big task but my own inclanation, at least long term would be to go with abc. Try this as an example from our folkinfo database. If you try the "transpose" link you will see we can even put a score or a MIDI out in any key.

We convert on the fly but another system may want to batch convert of maybe use some cahced combination of both.

The software I for the conversions use is abc2midi, abc2abc and abcm2ps and ghostscript. I think they will all run on Win, Mac and Unix like systems. The first 3 will even run on DOS. They are open source so compilations for systems where a binary does not exist is possible.

b) On the type of web based system I am thinking of, not significantly more difficult than compleing or ammending your Mudcat membership form.

c) It's not strictly web based in that you can run a web server on your own coputer purely to serve pages for your own personal use or within a LAN. The software I'm using is the so called LAMP (Linux, Apache, php, MySQL) but those are not rigid requirements, ie, it's likely that a Windows, Cold Fusion, MS SQL Server platform would be feasable if the idea was developed at Mudcat, etc.

The software I'm using could be set up on Windows, OSX on a mac and of course Unix like systems but I do not see it as being suitable for distribution if that is what you had in mind as I believe there would be far to many obstacles in the way of most "general computer users" to want to install the software (you know the web server, database server, etc.) to want to be bothered.

An idea I have for a possible futere distributable version as I said before would be to use the cross platform Java and probably HSQL as an embedded SQL database. It needs some research and time though and I don't know when I'd get round to it. If it seems like a good idea, perhaps someone else could take it up.

d) I can't remember how long it took but I took a few passes and trial and error to get there. I don't see why an Ask Sam conversion could not be made more automated though, although, I think you would be better off without it and the ideal, if possible, would seem to me a centralised eeb based point.

Anyway, I hope that helps and clarifies the lines I'm thing along. Of corse you could adopt parts (or none) of it if you desired.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Mick Pearce (MCP)
Date: 15 May 06 - 02:55 PM

That should have been: Try: this.

Mick

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Jon
Date: 15 May 06 - 03:03 PM

Thanks Mick. Don't know why I'm making so many messses with links in this thread...

Subject: RE: Tech: Digital Tradition Programmer Needed
From: dick greenhaus
Date: 15 May 06 - 04:40 PM

Jon-
I'm impressed. Is it possible to somehow indicate visually how the words synch with the played music (our present Mac and Windows versions do this, and I'm told my many people that it's very helpful)?

Also-
What kinds of processing time dows this system require for a search? It don't make no nevermind for a standalone system, but Max tells me it's of vital importance for Website operation.

The AskSam search capabilities are important to me, at least--a good part of the editing job is to locate duplications and slight variants, and the combination of full-text and defined-field searching is a great help.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: DMcG
Date: 15 May 06 - 05:22 PM

Dick, you asked Jon whether is it a difficult task to link each tune to a program (Noteworthy, perhaps) that both plays the tunes and displays the notes? for the folkinfo site.

It's not a general solution in the way that the PDF outputs are, but on my PC I have set up the 'abc' extension to be opened by an ABC player that will display the lyrics and music in the same way as the Spring 2002 download of the DT does. So if I click on Jon's 'abc' links I am straight into a playable lyrics+notes display.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Jon
Date: 15 May 06 - 05:29 PM

a) Not with what I'm using Dick. I think you would need to be looking at a Java Applet or perhaps Flash? to achieve that. It's beyond my scope as a programmer anyway.

I guess it would be something more to consider for a cross platform distribution version. Agian beyond my own programming ability but possible... and there are far better programmers than me around. I think one way or other something could be worked out...

b) This will sound contadictory but Max is right but it's never been a concern to me on a quiet (I think we get about 60-70,000 hits per month) site. To be honest, I haven't a clue, except to say things seem pretty instant here. And my PC never seems stressed in any way.

I'm going out for a game of pool now but what I'll try to do later tonight/tommorow is add some code to that dt search page to produce some times. You could then have a play with various searches and see how long each took on my PC which is running the folkinfo site but nothing else.

Maybe that would help to give an idea. But my PC (Athlon64 3000, 1GB RAM and SATA drive) probably is quite different to Max's hardware and his software is certainly different.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Bill D
Date: 15 May 06 - 06:35 PM

is Jeff reading this? Would his input be relevant?

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Jon
Date: 15 May 06 - 07:45 PM

OK Dick, it's a it crude but I'v put 2 times up. Build, the time it takes for php to make the SQL statement and Execute, the time it is taking to run the query and retrieve the result set. There are probably optimisations that could be made to speed things up a bit but this is how things stand.

Bill, if there was a chance of a Mudcat development, I would imagine Jeff's input would be very relevant.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: The Fooles Troupe
Date: 16 May 06 - 12:15 AM

"These days computers can download much larger files quicker"

Fallacy - 56 Kb dialup is still the same speed for those of us who may never get Broadband, what with being at the end of The International Super Goat Track...

BTW, the current HTML pages the Cat itself pushes out contain several Kb per page of unnecessary blank lines, tabs, and other stuff such as commented out lines that overall, must add up to quite a load slowdown on the server...

KISS!

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Jon
Date: 16 May 06 - 06:48 AM

Oh well Dick, for the fun of it, I've added what MySQL calls a BOOLEAN FULL TEXT (documentation at end of post) search this morning. It seems to be one of the quicker searches.

As you will notice, the queries do vary in time. All words being the slowest, I think because it can have to scan the song lyrics for each word. The worst case I've had on the searches I've tried is 0.2 seconds.

Things speed up considerably with a keyword combined with "all words" I think because it only has to search the words for songs with that keyword(s).

Natural and Boolean use a FULL TEXT index to search. I think that is more mySQL specific than the other searches. Perhaps the closet MC equivilant would be the Verity search.

Anyway, some bits extracted from the MySQL 5.0 documentation on Boolean Full Text Search Documentation:


The boolean full-text search capability supports the following operators:

+ A leading plus sign indicates that this word must be present in each row that is returned.

- A leading minus sign indicates that this word must not be present in any of the rows that are returned.

Note: The - operator acts only to exclude rows that are otherwise matched by other search terms. Thus, a boolean-mode search that contains only terms preceded by - returns an empty result. It does not return all rows except those containing any of the excluded terms.

(no operator) By default (when neither + nor - is specified) the word is optional, but the rows that contain it are rated higher. This mimics the behavior of MATCH() ... AGAINST() without the IN BOOLEAN MODE modifier.

( ) Parentheses group words into subexpressions. Parenthesized groups can be nested.

~A leading tilde acts as a negation operator, causing the word's contribution to the row's relevance to be negative. This is useful for marking noise words. A row containing such a word is rated lower than others, but is not excluded altogether, as it would be with the - operator.

* The asterisk serves as the truncation (or wildcard) operator. Unlike the other operators, it should be appended to the word to be affected. Words match if they begin with the word preceding the * operator.

" A phrase that is enclosed within double quote (") characters matches only rows that contain the phrase literally, as it was typed. The full-text engine splits the phrase into words, performs a search in the FULLTEXT index for the words. Prior to MySQL 5.0.3, the engine then performed a substring search for the phrase in the records that were found, so the match must include non-word characters in the phrase.

The following examples demonstrate some search strings that use boolean full-text operators:

'apple banana'

Find rows that contain at least one of the two words.

'+apple +juice'

Find rows that contain both words.

'+apple -macintosh'

Find rows that contain the word apple but not macintosh.

'+apple ~macintosh'

Find rows that contain the word apple, but if the row also contains the word macintosh, rate it lower than if row does not. This is softer than a search for '+apple -macintosh', for which the presence of macintosh causes the row not to be returned at all.

'apple*'

Find rows that contain words such as apple, apples, applesauce, or applet.

'"some words"'

Find rows that contain the exact phrase some words (for example, rows that contain some words of wisdom but not some noise words). Note that the " characters that enclose the phrase are operator characters that delimit the phrase. They are not the quotes that enclose the search string itself.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: IanC
Date: 16 May 06 - 07:04 AM

Er ... actually ... I think what most people are complaining about, when they say the DT's not been updated recently, is the fact that the inaccuracies which have been remarked on in 100s of threads over the past few years haven't been changed in the original.

I'm not sure how programming helps this.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Jon
Date: 16 May 06 - 07:23 AM

Ian, the DT is updated regulary. The delays are in getting Dick's stand alone AskSam version out in all the other (Windows, Mac, Mudcat online) versions out.

The suggestions I'm making are just some thoughts as to alternative methods of working that perhaps could address these issues.

If Dick could update and work on a live Mudcat version, problems at least with this site's copy need not exist at all.

One cross platform copy for distribution may be easier than Windows and Mac versions, etc.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST
Date: 16 May 06 - 07:27 AM

It's programming needed up front, to allow knowledgeable people (not the admins!) to go in and correct the inaccuracies as they come across them. That is the beauty of web based open source apps.

All software has to be adapted. The discussion here is, does DT and the accompanying post archive need to be converted to be made better as an application, not just 'upgraded' as in a bunch more song lyrics thrown in.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Jon
Date: 16 May 06 - 09:41 AM

The best search times BTW have included using the boolean search, eg. +wild +colonial has executed in 0.001 seconds. A "natural" search (whatever that is) for colonial boy has taken 0.0005 seconds.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Howard Kaplan
Date: 16 May 06 - 09:44 AM

Perhaps database programming should not be considered part of the DT group's core responsibilties. Perhaps, from the viewpoint of everyone except the DT editors, DT shouldn't have any software at all.

If all of the DT pages are put on a server as static ASCII text (i.e., simply as small files), and if the pages are organized with minimal HTML*, then it should be possible for all of the search capability to be outsourced to sites such as Google. There might be a DT site search form that automatically transmits the string which tells Google to search only DT instead of the whole web, but creating that form is much, much easier than actually hosting a search service. Google (or other search engines) can the do all of the heavy lifting, keep up with changes to the underlying platform software, etc.

(*Another alternative would be some combination of XML to mark fields and CSS to control their display. This could facilitate some kinds of search.)

The same process of making the pages clean will also aid those individuals who want to download the entire DT, run searches on their own computers, and view the results in their web browsers. These days, there are plenty of applications that let a user search a collection of documents on disk. Any user who wants such an application can download one appropriate to the user's platform.

DT's core competence is the compilation, editing, and presentation of song lyrics, tunes, and supplementary information. For this purpose, good software for the editors' use is essential. Behind the scenes, these editors need some kind of database to structure and validate the entries, keep the whole project organized, and retain their persoal sanity. Also, the database needs to be able to generate static web pages (including table-of-contents type pages and perhaps a few index pages) ready for upload to a hosting server, but such uploading can be done on a batch basis every few weeks. At current upload speeds between the database and the hosting server, even a 100 MB upload every few weeks would not be prohibitive; in fact, most pages would not change between uploads, so the actual updates could be much smaller. Also, a web site consisting entirely of static pages, with only a little scripting to facilitate submitting searches to outside sites, ought to be much more easily portable between hosts than is a web site with an active back-end database.

To summarize, yes, a DT programmer is needed, put perhaps not for the tasks originally envisioned.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Jon
Date: 16 May 06 - 10:02 AM

I don't think it is considered a "core responsability", Howard, except that Dick likes to make the DT available to others in a variety of formats.

Mudcat is just one location where the DT which is maintained by Dick can be viewed. Mudcat is however provides the main Internet presence, or home if you like. It is also the main collection point for additions and corrections to the DT. It is also a stite with existing database and (although I think there would be higher priorities at the moment) programming capabilites, incliding, as far as I know, a database containing all the dt material up to the last release from Dick.

The ideal on that side by my way of thinking would be a Mudcat development that Dick could use to input rather than a dt development.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Jack Campin
Date: 16 May 06 - 10:44 AM

I asked Dick about this a long time ago, when I was using a very old and slow Mac.

Macs don't *need* any indexing software. Given a great big collection of small files, the operating system's indexing utilities are all the database you need, and it's been that way for more than ten years.

So what I'd suggest is a version which simply has each song text and each tune in one file, with unique identifying tags. NO SOFTWARE AT ALL, just data. Maybe it could be minimally marked up in vanilla static HTML so web browsers could get into the act, but no Javascript or anything like that. And leave it up to the user what program they want to use to display the song texts.

I presume Microsoft is goýng to catch up wýth the idea of indexing theýr filesystems eventually, so this should be futureproof. No executable free-text database system can possibly be.

I last tried to create such a system on MacOS 9 - it was impossible because the export utilitýes tried to create directories wýth tens of thousands of entries, more than the system could cope with. But given access to the raw data it shouldn't be hard.

Again, PROGRAMMING IS A MISTAKE. DON'T DO IT.

And throw the existing software away. We don't need it any more.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Geoff the Duck
Date: 16 May 06 - 06:45 PM

Just a passing thought, not sure if there is any mileage in it.
One solution to making DT downloads platform independent might be to consider emulation. Is it possible to find an "old" OS which could run a workable Music/Text version of the DT (using existing software) but can be emulated on more than one platform.
Something such as the old Commodore Amiga runs as emulations on Mac, Windows, Linux and other systems (possible copyright constraints on distributing a working emulation - needs a rom image). I believe that the Atari is also emulated for Linux, PC and Mac.
You would have a set-up which might be older technology, but all you would need to do when a new version of Windows or Mac-OS is released is wait a short while until a new version of the emulator comes out (and the people who programme these sort of things seem to be very keen and dedicated to getting their emulators running on the latest set-up).
Another development to watch out for is portable emulators which do not need to be installed on your computer. Not much available yet, but watch the link..... (a lot of applications have been added since I discovered the Portable Apps web site).

Quack!
GtD.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: The Fooles Troupe
Date: 16 May 06 - 07:23 PM

An Open Source free 'thingie' that I have just heard of is Joomla.

It uses MySQL, and needs 'hosting' - but that doesn't have to be 'commercial'.

"Joomla! is one of the most powerful Open Source Content Management Systems on the planet. It is used all over the world for everything from simple websites to complex corporate applications. Joomla! is easy to install, simple to manage, and reliable. " - http://www.joomla.org/

"Joomla in Action
Joomla is used all over the world to power everything from simple, personal homepages to complex corporate web applications. Here are just some of the ways people use our software:

Corporate websites or portals
Online commerce
Small business websites
Non-profit and organizational websites
Government applications
Corporate intranets and extranets
School and church websites
Personal or family homepages
Community-based portals
Magazines and newspapers
the possibilities are limitless…

Joomla can be used to easily manage every aspect of your website, from adding content and images to updating a product catalog or taking online reservations."

That's enough - interested parties read for yourself.

But don't forget - 'cutting edge' solutions can cut those who partake early... :-)

The DT 'enterers of data' don't need to know much about this - the 'techie maintainers' do that.

I do like Howard Kaplan's suggestion above of just pure ASCII or HTML pages though - KISS. Joomla seems (to be intended, anyway!) to allow easy filing, indexing and access to things like the words, mids, etc as well.

Robin

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Jon
Date: 17 May 06 - 09:19 AM

I'd stll favour a live database solution for part of the problem but maybe that's just me.

Anyway, I was up very late last night and decided to have a play with my dt database, first of all trying a to produce a large single text file and later, muliple html files.

It's a bit crude again but I can produce this from the mySQL database. On my PC, it can generate all the html files in 40 seconds.

It's on my "general purpose" PC which I do switch off and sometimes boot up in Windows, etc but I don't really want to transfer the 13656 files over to my "production PC".

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Mick Pearce (MCP)
Date: 17 May 06 - 11:18 AM

Here are some of my thoughts on the maintenance of the DT and contributions via Mudcat.

I think the DT needs to keep a central editor/editing team to maintain the definitive version of the DT.

However, in Mudcat instead of the ad-hoc adding of lyrics we have at the moment and users having to search throught the forum for songs not in the DT, why not have an actual Add Song form with all the appropriate fields to add the song to the online database, giving it a DT number. The index only needs a flag to indicate that it's not part of the release DT yet, but can be included in the browse and search DT functions. This will also make it easier for the editors to locate the new songs and produce the final release version later, flagging it as a release file.

Similarly corrections to the DT could be linked directly to the songs in the DT, including the provisional entries added by users. This could be just a link in the song entry pointing to the message or thread, but having it in the database will also make access easier for the editors later. In view of the fact that new additions often
have several corrections more or less immediately, it might be wise to allow the person who initially adds the words to be able to update them (possibly for only a limited time).

Tunes posted could similarly be linked directly to the song via a proper form for submission.

These options would allow the for the openness that some users have been requesting (a feature which is currently there but ad-hoc via the forum) while leaving final control in the hands of the editing team.

It should also make new releases of the DT easier to maintain, since you could do an incremental update of changed/added songs only.


As for the released version of the DT, I quite like the idea of a browser-accessed version. As Jon has shown above it takes nothing to generate the songs as HTML files (It's possible to clean them up a bit too - in the version I imported into Access, I had a program classify the various line types - title, song text, attribution, source, date, submittor, keywords etc. This information can be used to systematise the html).

Also if you generate midi song files suitably you can use a karaoke style player to play the tune with synchronised words. (abc2midi generates a suitable file if the w: tags are used in the abc for example). A java-based appelet would let you have platform-independence for this. While I do like the current player, having the printed music doesn't need to be tied to the karaoke style display - users of the printed score are less likely to be interested in the karaoke player and vice-versa.

That does only leave the printed score to deal with. While I can do that easily via postscript, the general user probably doesn't want to go to the trouble of installing Ghostview/Ghostscript or some similar postscript viewer, so I don't have a better solution for this at the moment.

By way of sizes related to tune files, I have just under 4000 tune files; the abc is about 11Mb, 4Mb zipped (these contain the full song text as well as the synchronised verse); the midi is about 24Mb, 5Mb zipped; the postscript (again including full text) 128Mb, 35Mb zipped. Based on the html for ARDTAC from Jon's site (around 3Kb on my machine), the 8000 or so text files should generate about 20Mb unzipped (and if I had to guess around 5Mb zipped - I haven't got enough to make a decent estimate).

I'm not saying this is the way to go, just offerning it up for thought.


Mick

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Mick Pearce (MCP)
Date: 17 May 06 - 11:20 AM

I should have added that if you have the abc2midi generated files from abc with synchronised text, they will display karaoke style with eg JetAudio or VanBasko's karaoke player.

Mick

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Mark Clark
Date: 17 May 06 - 12:24 PM

Interesting.

I'll add a few thoughts that don't necessarily add up to a solution. Feel free to ingnore. By way of background, I became a professional programmer in 1967 and though I'm now retired I've written for most platforms from microcomputers to mainframe behemoths. I'm familiar with database theory and practice as well as Internet protocols and WWW programming. I've used several assembly languages, FORTRAN, COBOL, PL/I, C, C++, SmallTalk, Java, LISP, Perl and a few others. I'm familiar with PC-DOS, Windows, Unix/Linux, NeXTStep (predecessor to Mac OS X).

I looked at the AskSam Website and my impression is that the product is entirely appropriate as an engine for the DT. It is widely used in many situations, kept current by its authors and seems, as Dick has said, particularly well suited for our purpose. Unfortunately, I've been unable to determine, from the site, what platforms AskSam targets. Can anyone tell me whether the professional version of the AskSam product is restricted to certain operating systems?

The idea behind Wiki is that everyone can add to and edit the information being hosted. In many implementations, such as Wikipedia, additions and changes are subject to review prior to open publication. And even though Wikipedia contains errors, it's error rate has been on a par with that of The Encyclop�dia Britannica. But there is a huge amount of work required to keep Wikipedia updated and current.

A Web interface is probably the most appropriate choice for the DT. Many companies and universities distribute information on CDs that relys on the end users' Web browsers for the user interface. I don't believe there are any useful computers available today that lack a Web browser. The trick for the developer is to maintain the dicipline to use open standards over proprietary extensions.

Jon's use of the ABC Plus tools to generate pages is a great idea and places very little load on either a host or user's system. ABC Plus (abcm2ps) supports the alignment of lyrics with the score but this requires intimate knowledge of the song and must be manually coded for each tune. I don't think it's likely that an accurate, automated lyrics alignment algorthm can be constructed within the scope of the DT project.

I don't think there is anything that makes the display of an ABC score incompatible with AskSam. It just requires that the database include the ABC code and that AskSam provide an API (application programming interface) such that database querries can be passed to the application scripts. As a widely-used professional tool, I can't believe that AskSam doesn't provide the needed APIs. Of course the DT must support entries that include nothing but lyrics as well.

Dick and Susan have maintained the DT for, what?, twenty years now? I seem to remember a floppy disk distribution available by mail. And wasn't the DT hosted at Xerox (PARC?) for quite a while? My impression has always been that some very bright and forward thinking folks worked on the DT. It shouldn't be thought of as some ill-considered, half-baked kludge.

As the DT has been open, so is software development open. SourceForge.net is a resource used by open source developers around the globe for collaboratively working on software projects. I think it's common for software developers to be musicians as well and I'll bet there are developers, perhaps unaware of the DT, who would be willing to undertake Dick's project. SourceForge provides the mechanisms for collaborative work so many developers could be involved and contribute as they are able. This would still allow Dick and Susan to direct the project and shape the result.

As much as I admire all the work that Max has done, I've been puzzled as to why he continues to own and maintain the servers needed to host The Mudcat Caf�. Professional hosting is available for all levels of need that may be had at a price far cheaper than doing it one's self. Hosting companies such as Lypha, a subsidiary of Tucows, offer cheap hosting with high bandwidth, high storage, database hosting, Web scripting, maintenance tools, etc. These services virtually never go down and backups may be automated. They offer both Linux (preferred) and Windows OS support. They also collect usage data for planning purposes and are nearly invulnerable to hostile attacks. Max is a very smart guy so I assume there is some reason why he prefers maintaining his own servers, I just haven't seen the rationalle explained.

As I said, just some thoughts.

      - Mark

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Jon
Date: 17 May 06 - 02:00 PM

Mark, as far as I know Ask Same is currentl Windoes only (the FAQ suggested using Virtual PC for a Mac) and I believe Dick's DOS version at any rate is stand alone. The DOS version does have a free (I've a feeling that Dick negotiated with them many year ago) runtime to read the proprietry .ask file but I believe any other distributed version involved either involves works on it's .ask file format or on an ASCII export to get the data into whatever usable format for the program in question.

While it may offer great speed, etc. (and we may disagree), I see all the above as drawbacks compared to an SQL database. My (OK perhaps wrong) feeling is that its use has contributed toward the long time between new releases of the dt.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: dick greenhaus
Date: 17 May 06 - 04:28 PM

I've asked Max to check out Jon's site, and give me some ideas as to whether or not this approach would work with the hardware that's in place. Once we get a clear idea of exactly what constraints we have to work within, we can proceed to begin.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Jon
Date: 18 May 06 - 07:31 AM

Dick, I hope something useful somes out of some of the menay suggestions kicked around in this thread.

-------
Mark:

Professional hosting is available for all levels of need that may be had at a price far cheaper than doing it one's self. (etc.).

I'm not convinced you can generalise that much.

I don't think my home hosting where the costs are electricity and a static IP address (I'd have ADSL anywat) that shared hosting I have/do use and some allowance for hardware replacement over time is that different. Folkinfo (although I had rather more than I needed) cost me more on the resellers account I used to use.

Both seem to have plusses and minuses.

Having now used both, on the plus side, I have far greater stability than I have had with shared hosting. This is perhaps in part due to a Linux vs Windows (I've not used Linux share hosting) thingy but also part due to not being at the mercy of other users, some of whom can go way over thier allocations (until caught) or a plain clumsy - something as simple as putting an asp script and getting the program stuck in a loop because say they forgot a MoveNext in a database operation can over time kill every user - even non asp users on the server. I also feel I have that little bit more freedam.

On the minus side, we have the what happens if something goes wrong questions. One (although since starting folkinfo, I have never had to make an unplanned reboot) the simpler what if it went down and I'm not around thought and on a more serious note, would I be as equipped as a professional host in the even of an all out DOS or other attack?

I think where costs would really tip the ballance for me is if I did something that outgrew my 256K upload limit. The costs here say for SDSL (if I could get it even) as far as I can make out would cease to make home hosting a viable option.

An alternative, to shared or dedicated hosting, I might also onsider in such a situation is co-loation. I've seen prices like prices starting at £25 pm for 1U with 100GB bandwidth in Mancester...

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Jon
Date: 18 May 06 - 07:35 AM

"since starting folkinfo", transferring to home hosting that is.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Mark Clark
Date: 18 May 06 - 12:15 PM

Jon, I have two small Web sites at Lypha right now and have found them to be extreemly solid. They really never go down and for US$6.95/month (two year contract) you get 50GB of storage and 400GB bandwidth. Here is a list of what is included. Here is a link to descriptions of their communications infrastructure. You'll see that no home-based installation could reasonably compete with that.

In the US there are very many local hosting companies who are selling time on somone's back room PC farm and it's hard for non-technical people to know what they're getting into just from the sales material. One good resource to check on the viability of hosting companies is the Netcraft site. They constantly monitor every aspect of site technologies and hosting performance.

Personally, I would never build any Web site on Windows technology. I actually like Microsoft and use their products at home but the Internet and Web really run on UNIX/Linux. Microsoft's share continues to fall and they never really won in that arena. The preference for UNIX/Linux worldwide has little to do with software costs but is largely due to performance and security issues.

I haven't seen any hit and data transfer stats for Mudcat so I don't know what the Lypha rate might be but it would be interesting to see a comparrison.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Jon
Date: 18 May 06 - 01:00 PM

I think I stand corrected Mark... One could do a lot with that!

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Mark Clark
Date: 18 May 06 - 05:39 PM

Also, as a technical solution, the Digital Tradition could be hosted someplace like Lypha on a shared or dedicated "WinTel" system but without allowing any direct user access. Instead the DT and AskSam would be set up purely as a server hosting a "well known" Web service. (Some of the developers here will know what I'm talking about.)

Then the Mudcat site, or any authorized site, however hosted, would simply use Web Service technology to query the DT. This will keep the DT and Mudcat searches from impacting each other but still provide for a seamless integration of the two.

The Web Service integration could be scripted so the DT call really comes from the users' systems, not Mudcat. That way the DT transfer wouldn't eat up Mudcat bandwidth.

Think about it.

      - Mark

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Mick Pearce (MCP)
Date: 22 May 06 - 03:54 AM

I was walking the dog just now and thinking, as you do, about DT searching.

When I imported my version of the DT into Access I had some code classify all the lines in the songs: Titles, Song Text, Notes, Laws, Child etc. (It wasn't perfect - Notes were sometimes confused, but I had a little browser that let me skim through the songs and then reassign the odd ones). I was thinking that with a little more thought I could have subtyped Song Text into First Line, Chorus and/or Refrains. This would allow searches to be made more specific for First Line (as the distribution version of DT allows), Chorus lines or Full Text (and Notes if you wanted to search that separately). Given that people often know the first line or chorus of the song they're looking for being able to search on these alone would be quite a saving:

In my db the figures are something like:

    8,000 songs
380,000 lines everything
305,000 lines song text (inc blank lines, so assuming 4 line verses
                           about 240,000 non-blank lines)
   23,000 lines of notes

So the searches would be of the order:
    8,000 lines for titles
    8,000 lines for 1st lines
   32,000 lines for chorus seach, assuming each song has a 4 line chorus

All of these a lot smaller to search than the full text, which you could save as a last resort.

Mick

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Mick Pearce (MCP)
Date: 22 May 06 - 04:12 AM

A crude look at my database just selecting non-blank song text lines with 1st 2 chars blank (the choruses are usually indented) or "ch" (for Ch., cho. and chorus, which sometimes appears on the 1st line of the chorus) yielded just under 36,000 lines, some of which are not actual chorus (Chorus: appears on line before the chorus text, some songs are actually indented, and probably some choruses are not indented). But even this crude assignment of lines would probably find most chorus searches before needing to check full text.

Mick

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Jon
Date: 22 May 06 - 04:50 AM

I suppose it could be done if searches proved too costly but I'm not convinced there would be a need.

On my PC, of my experiments in different searches, "boolean" searches based on a "fulltext" index are typically executing in 100th-1000th of a second. The slower searches wich do search the song text itself are taking 1-2th's of a second.

I don't know how Access might cope though - it's a nice desktop app but not a high performance database.

BTW, structure, I only split "song" into SongID, Title, Song and Filename but it would have been useful to "split" further. I've 4 more tables, tune which currently holds the tune in Songwrite format, keyword, songtune and songkeyword.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Geoff the Duck
Date: 22 May 06 - 04:55 AM

Just a comment on first lines. When I hear a song performed, particularly when unaccompanied, the singer often starts singing when my brain is not tuned in. It often takes until the second line before I am actually aware what is being sung. As a result, I do not actually know the first line of the song.
I have often thought that listing by second line or last line would make a song lyric easier for me to find.
By the way - I was very impressed by the speed which Jon Freeman's rough demo gave displays of song lyrics and ABC versions of the tune.
Quack!
Geoff.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Mick Pearce (MCP)
Date: 22 May 06 - 07:47 AM

When I suggested adding a chorus search I wasn't only thinking of machine efficiency, but also of user efficiency in terms of number of results returned. For example, I just did a search for barley which returned 50-odd songs from a full search but only 14 from a search restricted to an approximation to chorus lines.

Mick

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Susan of DT
Date: 22 May 06 - 10:25 AM

Full text search is the first resort, not the last
With a full text search you don't need to know what "someone" thought the title was or thought was the first line. You can find a song by a unique phrase, whereever it appears in the song. The current search on mudcat is not the best search - it does not track the "little" words, so if your phrase contains one of these, it does not find it. Our search works better.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Mark Clark
Date: 22 May 06 - 12:06 PM

In my experience (35+ years), software developers will usually favor solutions with which they are already familiar. They also gravitate towards solutions that simplify the work of development and maintenance, often without regard for the end user of the software. Once developers have had significant positive experience with particular technologies—database, language, platform, etc.—that technology becomes the basis for all their modeling thought processes.

My observation has been that Dick and Susan have many years of thought and experience behind them and maintain a solid understanding of the ways in which the DT is used and the search styles that best support the intended audience. I can see no advantage to shoehorning the DT into a relational database paradigm except to provide developers with a more familiar tool set.

High speed full text indexing and searching is a mature technology. NeXT Computers (NeXTSTEP OS) in 1990 came with something called the Digital Librarian. This automatically maintained full text indexes for every file on the system and allowed the user to find any file in the file system instantly on the basis of content alone. This on a processor operating at 33 MHz, one tenth of today's processor speeds.

NeXT's Digital Librarian became the basis for the Internet search facility known as The Gopher, widely used before the advent of the World Wide Web. It's interesting to note that Tim Berners-Lee used NeXT computers to develop his original World Wide Web servers and browsers. Recent Microsoft operating systems also come with a full content indexing option.

I have no experience with AskSam technology but, reading their site and seeing how it's used, I'm guessing that the developers have incorporated the most advanced full text indexing and search algorithms available. I don't think I'd dismiss AskSam as outdated technology.

I think the key for DT development and growth is to isolate the DT on a dedicated WinTel server hosting only the DT and AskSam. There would be no direct user access to the server. Rather a network communication protocal known as Web Services would be employed to give client systems (which can be other servers or end users' systems) the ability to maintain and query the DT without placing any load on the local system. The DT server, then, can be sized to provide the desired level of performance based on the demand.

This solution also means that the Internet DT "service" will have searching capabilities identical to the version distributed on CD-ROM. Web sites such as Mudcat would use Web Service technology to submit queries to the DT but this will be invisible to the Mudcat user. There need be no change in Mudcat's apperance, user interface or performance. In fact, the DT Web Service could be physically located inside Max's installation if that is deemed best. I tend to think an externally hosted solution would offer greater power and flexibility at lower cost but I probably don't have all the details needed to say that with absolute certainty.

      - Mark

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Jon
Date: 22 May 06 - 12:52 PM

I'm not doubting AskSam's ability as a search engine. What I am questioning is the following.

Will Dick's version allow web/ network searches? As far as I know, it is stand alone with no networking abilities.

Having looked again, there is a reference to a network version but I haven't been able find any more on it - maybe that would do it? Aloso there is a web publisher version? Maybe that would do it?

Distribution, I see there is an free Ask sam version for Windows but what about other platforms, if you wich to keep the same look and feel as Dicks version?

It's these sorts of doubts and issues that tend me towards using an iddustry standard SQL database, some of which, including MySQL do have full text searches.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Jon Freeman
Date: 22 May 06 - 01:01 PM

Ah, re the networking version, I see against the AskSam professional version priced at $395, it says:

"For Network options and pricing please contact our sales department at 800-800-1997 or email us...".

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Mick Pearce (MCP)
Date: 22 May 06 - 01:03 PM

Susan

The DT currently supports title searches, so if "someone" can have an idea of what is in a title, why not the more obvious what someone thinks is in the first line or what someone thinks is in the chorus. This is an end-user's idea not a developer's idea (in my 35+ years as systems programmer, systems designer and systems manager I've always designed from the end-user's perspective not the ease of implementation). To quote my example above, if I was looking for a song that I thought had barley in the chorus I'd prefer to look through the 15 songs with barley in the chorus rather than 60 songs with barley in the song. I do listen to songs and I know what a first line is and what a chorus is, usually more reliably than what a title is; if I didn't find it from what I thought was in the first line or chorus, then I'd be willing to look at the list from a full-text search.

Saying that full-text search is the first resort is not the best service you can give a user (certainly not this user).

As regards the delivery method I have no problems with whatever system (I use the current download version all the time at home) is used to do that (unless I had to implement the system). My problems would be with the data organisation. Lumping everything into the text is not good (Try searching for Irish Ways And Irish Laws if all you know is that the song had the word laws in it -see what a full-text search gets you then).

Mick

Subject: RE: Tech: Digital Tradition Programmer Needed
From: MMario
Date: 22 May 06 - 01:09 PM

a full text search would include second lines and choruses, would it not? and titles.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Mick Pearce (MCP)
Date: 22 May 06 - 01:28 PM

Yes, but all at the same time. The point would be to search them separately.

Mick

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Jon
Date: 22 May 06 - 01:35 PM

a full text search would include second lines and choruses, would it not? and titles.

I suppose it depends on the usage of fulltext. I suspect I have been inconsistant but have mostly been using it to refer to the mySQL fulltext searches. In my case, there is an index built up from Song Title and SongLyrics and the relevant searches search both columns but I could just as easily have a fulltext search on just one column.

Susan, I don't know about the mudcat search but the mySQL fulltext searches, the search simply ignores certain words, eg. a search for "The Wild Rover" would only search for "Wild Rover". The fact it doesn't use the word "The" doesn't prevent "The Wild Rover" being found but a search for "The" would yeild no results. I would suspect the Mudcat one works the same way.

With MySQL at anyrate, such things are customisable but mine is left at the default list of "stopwords" and has a minimum word length of 4.

As for backgrounds, seeing as they seem to crop up a few times. I have never worked professionaly as a programer, analyst, etc. The clostest I ever got to computers was when I had a job entitled "systems co-ordinator" for production control with a white goods company. I would say the users side was important to me.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Mick Pearce (MCP)
Date: 22 May 06 - 02:06 PM

Just thought I'd better add that I'm not trying to detract from the DT as it currently is. Dick and Susan have provided a great resource and as I said I use my downloaded version often. As with Mudcat itself, I find no grounds to denigrate a resource that someone has taken the time and trouble to make freely available for no reward.

I do have it here also in a modified form (Access database and a MySQL mirror) because I wanted to use it to generate an abc version of the tunes and not wanting to purchase AskSam I copied it into a database I did have and took some time to do a bit of reorganisation and expansion of the tables. I also generated Parson's coding from the tune files, but again to make that really useful, I'd want to separate out verse and chorus in the tune files, which I might get round to some day.

All the thoughts I've mentioned above are only ideas I have. I'm happy with whatever direction the DT goes.

Mick

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Mark Clark
Date: 22 May 06 - 02:49 PM

Jon wrote:

Will Dick's version allow web/ network searches? As far as I know, it is stand alone with no networking abilities.

and

"For Network options and pricing please contact our sales department at 800-800-1997 or email us...".

I don't think you understand what I'm proposing. For what I'm proposing, there is probably no need for "networking abilities" in the AskSam product. Typically, networking features in such a program allow it to act as a server over a local area network (LAN). All that is needed for my proposal is the abiliity to serve multiple local queries at the same time or allow multiple instances of the server to run simultaneously. All the Internet (networking) communication would be handled by Web Service technology that would simply pass queries to AskSam and rout results to the requesting client. Web Services are robust and loosly coupled making the technology ideal for Internet use.

I haven't reviewed AskSam's licensing requirements. It may be that they require "network licensing" to be legal for multiple simultaneous queries even though network communication is not needed.

      - Mark

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Jon
Date: 22 May 06 - 03:09 PM

You are right, Mark, I had not understood you.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: dick greenhaus
Date: 22 May 06 - 08:21 PM

I really appreciate all the input. I'm chewing on some of the ideas as I sit here, and I'll be discussing implementation with Max this weekend. BTW, for those interested, AskSam--unlike other full-text-search databases--permits the use of defined fields, so that there's no problem (aside from entering a few keystrokes for each entry) in having separate searches for titlesm or choruses, or third lines or whatever.It just hasn't proven to be particularly helpful.
    The difficulty with search engines that ignore the "little" words emerges when you search for a particular phrase, rather than a bunch of separate words. This has been a major headache in the HTML implementation at Mudcat. Another useful feature that most databases seem to lack is the ability to use wildcards--symbols that can represent either any single character (? is used in AskSam) or any series of characters (* will stand for any number of characters up to the next space). So, if you're not sure whether the word you're looking for is "bonny" or "bonnie", entering "bonn*" will return lyrics containing either. And, in AskSam, wildcards can be used at ehe beginning of a word or phrase as well as at the end.
    If it turns out that AskSam is the way we decide to go, there will be no problem using it as an Internet application.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST
Date: 22 May 06 - 09:05 PM

Another useful feature that most databases seem to lack is the ability to use wildcards

It's a standard SQL feature, Dick. See here for an example of how it's used in a query (SELECT statement).

I think some of what you see or don't see comes from us trying to provide a simple user interface and perhaps not going as far as we could in building up a query.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: DMcG
Date: 23 May 06 - 08:09 AM

That last Guest (Jon?) forgot to post the statement itself. Here's some MySQL documentation on pattern matching.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Jon
Date: 11 Jul 06 - 07:59 PM

Might as well refresh this one - see if anything has happened...

I had a play with it over the weekend for my own ammusement. I know I have been given the abc for the current version but wouldn't have straight away for any new version so I thought I'd try my hand at an sw2abc php script and feed the results of that to abcm2ps, etc.

It needs a lot of work yet (if I get round to it) but it's getting there on some songs like this one are working. It will fail on multivoice, it's treating slurs as ties, some triplets are odd, and I don't know what else.

I know of Yet Another Digital tradition BTW but I think he goes to PMW (he used to go to Lilypond) for the graphics rather than abc and his abc conversions he offers don't seem to have words.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: dick greenhaus
Date: 11 Jul 06 - 08:51 PM

Guest Jon-
Is it feasible to do what you've done under program control? I like what it looks like, but I find manually converting almost 5000 tunes a bit daunting.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Jon
Date: 11 Jul 06 - 09:00 PM

Dick, these are just converting from your Songwright to abc (and then on to other formats) when a page is requested.

Mick Pierce may well already have a fully functional (which mine isn't) script/program that would do the lot in a batch job - I'm not sure.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Mick Pearce (MCP)
Date: 11 Jul 06 - 10:04 PM

My conversion was actually done in Access (I was putting the DT in there) so it's essentially VB procedures. I still have it set up and if you want I'm happy to convert the SW files (it was about 4000 files when I last converted it) - it's all automated for a batch run. (Similarly for midi and postscript after that - trivial shell batch files using abc2midi and abcm2ps).

There were some problems files in the last lot (I have a list!) - there were a few odd files that weren't SW. I also didn't convert the multivoice files; there were literally only a handful and I didn't think it worth the extra programming effort and they could be done manually.

(The conversion started from the sw2abc awk script, though my final conversion program is considerably longer, but more thorough!. Jon if you're interested I still have the sw2abc script - it's about 300 lines or so).

Mick

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Jon
Date: 11 Jul 06 - 10:32 PM

Mick, I found that script too BTW and modified and used part of it for my php play. I might as well post that part here as it could be useful to others, regardless of language used.

      //rests
      $pitches["r"] = "z";  //currently using the same rest
      $pitches["R"] = "z";
      $pitches["x"] = "z";
      $pitches["X"] = "z";
      
      //notes
      $pitches[":"] = "A,,";
      $pitches[";"] = "B,,";
      $pitches["<"] = "C,";
      $pitches["="] = "D,";
      $pitches[">"] = "E,";
      $pitches["?"] = "F,";
      $pitches["@"] = "G,";
      $pitches["A"] = "A,";
      $pitches["B"] = "B,";
      $pitches["C"] = "C";
      $pitches["D"] = "D";
      $pitches["E"] = "E";
      $pitches["F"] = "F";
      $pitches["G"] = "G";
      $pitches["a"] = "A";
      $pitches["b"] = "B";
      $pitches["c"] = "c";
      $pitches["d"] = "d";
      $pitches["e"] = "e";
      $pitches["f"] = "f";
      $pitches["g"] = "g";
      $pitches["h"] = "a";
      $pitches["i"] = "b";
      $pitches["j"] = "c'";
      $pitches["k"] = "d'";
      $pitches["l"] = "e'";
      $pitches["m"] = "f'";
      $pitches["n"] = "g'";
      $pitches["o"] = "a'";
      $pitches["p"] = "b'";
  
      //accidentals;
      $accidentals["-"] = "";       //none
      $accidentals["&"] = "_";      //flat
      $accidentals["%"] = "=";      //natural
      $accidentals["#"] = "^";      //sharp
      $accidentals["$"] = "^^";     //double sharp
      $accidentals["*"] = "__";     //double flat
  
      //durations in 16th notes
      $durations["1"] = "16";
      $durations["2"] = "8";
      $durations["3"] = "12";
      $durations["4"] = "4";
      $durations["5"] = "6";
      $durations["6"] = "1.66666666666666666666";      //quarter triplet
      $durations["7"] = "1.33333333333333333333";      //eigth triplet
      $durations["8"] = "2";
      $durations["9"] = "3";
      $durations["0"] = "1";
      $durations["-"] = "1.5";
      $durations["="] = "1/2";
      $durations["/"] = "5.33333333333333333333";   //triplet???
      $durations[")"] = "0.66666666666666666666";   //sixteenth triplet
      
      $codes[" "] = "";  //normal
      $codes["_"] = "-"; //tie
      //
      $codes["+"] = "[]"; //chord
      $codes["."] = ".";  //staccato
      $codes["-"] = "";   //legato
      
      //header fields
      $headers["N"] = "T:";
      $headers["C"] = "C:";
      $headers["A"] = "S:";
      $headers["T"] = "Q:";  //not clear on sw T/S relationship
      $headers["S"] = "I:";
      $headers["K"] = "K:";
      $headers["B"] = "M:";
      $headers["F"] = "N:";
  

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Jon
Date: 11 Jul 06 - 10:45 PM

Found the websites - Steve Allen's sw2abc. Some additional SW information

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Mick Pearce (MCP)
Date: 12 Jul 06 - 07:52 AM

Incidentally I've been developing a Java browser for the DT, keeping the data in an Apache Derby database (a small footprint Java relational DB). The browser and search facilities were almost trivial to set up. The advantages of Java are that it's completely transportable across platforms (well maybe not DOS; but is anyone using the DOS version of DT?) and that the apps are very small.

I'm running Derby as a desktop app at the moment, but it can also run in server mode with only a minor change to the program (load a different driver). Also it's trivial to make it handle other databases - MySQL, Access, ODBC - it really only needs the program to know which database driver to load.

The data structure keeps my preferred semantic distinctions - ie song text separate from notes and identification fields.

I'm also adding a song entry feature so that you can keep your own songs and tunes in the same database. Planned features are to hold abc tunes in the database and to keep links to external tune/music files. I'm also implementing an abc2.0 system (the spec has remained in draft for several years now and no work is being done on it as far as I know. It must also be said that there are some problems with the spec that I would change - macros in particular aren't well thought out, but I believe there ought to be some standard to use.), also in Java, intending ultimately to generate PS and use the Toastscript (Java) PS renderer to display the tunes. Sadly work has to stop until mid-August now as I have other things to do, but I'll keep you posted.

Mick

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Jon
Date: 12 Jul 06 - 08:28 AM

That sounds great Mick! Exactly the sort of distrubution that's needed IMO.

I think the main reason for a DOS version is that it and it's AskSam engine has been better for searching than the other releases.

Must take a look at Derby sometime - I'd not heard of it.

Re abc: I would be using abc 2.0 too. I wish it would hurry up to a standard but I'm not holding my breath.

It's a shame but the business with aligning words remains a problem, as different programs have their own ways. At the moment, I take the view if it works with abc2mps it is correct.

I'd hope that if the dt really does prove to be serious about using abc, it's presence in that form might actualy provide the clout needed to get these minor, but irritating for those who want to distribute abc with words and be (at least reasonably) sure it will work with any current program, problems out.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Wayne Farmer
Date: 18 Oct 07 - 03:07 AM

My two cents, after discovering this thread and reading much of it:

Among talk of standards for the Digital Tradition database, I didn't see mention of the MusicXML standard. As of 2007, MusicXML would seem to be a strong candidate for a standard.

Wayne Farmer

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Geoff the Duck
Date: 18 Oct 07 - 05:13 AM

The format seems to address the alignment of words and melody for a printed output. Presumably the source XML file can be re-edited if the tune needs to be modified. Looking at examples of a score compared with the XML file, it looks as though it codes each vertical "slice" of the stave as a separate entity, so could be difficult (although possible) to edit by hand.
It is designed to be produced and read by purpose built programmes, although if the format is open source, there may be free progs which will do it. If it is becoming a format supported by a range of music editing programmes, it would definitely be worth investigating.
Here is a page of links to rogrammes which can use Music XML BLICKY.
This one musicrain seems to be an elegant solution for displaying notation, words and audible tune online, as long as you have a browser running recent version of Flash Player. Look at the Demos - transposable and printable score which also play as music. I couldn't find any mention of how much it costs though.
Quack!
GtD.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: dick greenhaus
Date: 18 Oct 07 - 09:48 AM

Geoff-
The musicrain link doesn't seem to be working right--can't see any examples.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Jeri
Date: 18 Oct 07 - 10:12 AM

The 'musicrain' link works for me, but after a few minutes of converting, I didn't want to wait any longer. 'Elegant' may mean effective but impractically LARGE.

Dick, you may have an older version of Flash. Opinion: software that requires users to have other, specialized software isn't going to be a good solution. Don't know nuthin' 'bout no programmin', but I got plenty of opinions about usin'.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST,Jon
Date: 18 Oct 07 - 11:05 AM

I believe musicXML is useful in providing a common format that can be imported to and exported from a good number of music programs (see http://www.musicxml.org/xml.html) and that it can represent complex scores but I think it's OTT for the dt. See http://www.musicxml.org/xml/helloworld.htmlfor their "hello world" example.

Technically, I think Songwrite is adequate for the dt. The problem with it is that it is an obsolete proprietary format. The closest match to what is done in Songwrite in the dt I know of that is current and supported is abc.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: Geoff the Duck
Date: 19 Oct 07 - 03:51 PM

Dick - On the left edge of the picture, there is a semi-circular bit missing which comntains text saying "Try It Out" and underneath "Live Demo". Clicking there runs a bit of javascript that opens a pop up window containing a selection of different types of music for it to process.
The software it runs may be too slow to be practical on an old slow computer, but it shows what can be done and may be worth noting for future reference.
I've just tried the Elite Syncopations demo and noticed that it doesn't play repeats, it moves from last bar of repeat 1 to the last bar of the second time through.

Jon - Yes, I agree that Music XML is capable of way beyond what is needed just to represent a simple melody line, but if it is going to become the industry standard file exchange format, it might be worth adopting to future proof the Digitrad. It might be worth looking for    software which can batch convert existing files to the new format.
Quack!
Geoff.

Subject: RE: Tech: Digital Tradition Programmer Needed
From: dick greenhaus
Date: 19 Oct 07 - 06:46 PM

Jon-
On your site, you display the scores; presumably from an ABC to score program. Do you have an automated approach for converting SongWright files to ABC?

Subject: RE: Tech: Digital Tradition Programmer Needed
From: GUEST
Date: 19 Oct 07 - 07:56 PM

I don't know Goeff. I understand your reasoning but I think I would go the other way and believe abc will be around for a long while.
-------
Dick. I think Mick Pearce has a much better job but, yes, I do use a script, something I hacked together which seems to get most dt files I tried into abc when a page is requested. I doubt it would be of much help but you or anyone else are welcome to do whatever you like with that.

---------

  sw2abcdef.php
        //rests
      $pitches["r"] = "z";
      $pitches["R"] = "z";
      $pitches["x"] = "z";
      $pitches["X"] = "z";
      //notes
      $pitches[":"] = "A,,";
      $pitches[";"] = "B,,";
      $pitches["<"] = "C,";
      $pitches["="] = "D,";
      $pitches[">"] = "E,";
      $pitches["?"] = "F,";
      $pitches["@"] = "G,";
      $pitches["A"] = "A,";
      $pitches["B"] = "B,";
      $pitches["C"] = "C";
      $pitches["D"] = "D";
      $pitches["E"] = "E";
      $pitches["F"] = "F";
      $pitches["G"] = "G";
      $pitches["a"] = "A";
      $pitches["b"] = "B";
      $pitches["c"] = "c";
      $pitches["d"] = "d";
      $pitches["e"] = "e";
      $pitches["f"] = "f";
      $pitches["g"] = "g";
      $pitches["h"] = "a";
      $pitches["i"] = "b";
      $pitches["j"] = "c'";
      $pitches["k"] = "d'";
      $pitches["l"] = "e'";
      $pitches["m"] = "f'";
      $pitches["n"] = "g'";
      $pitches["o"] = "a'";
      $pitches["p"] = "b'";
  
      //accidentals;
      $accidentals["-"] = "";       //none
      $accidentals["&"] = "_";      //flat
      $accidentals["%"] = "=";      //natural
      $accidentals["#"] = "^";      //sharp
  
      //durations in 16th notes
      $durations["1"] = "16";
      $durations["2"] = "8";
      $durations["3"] = "12";
      $durations["4"] = "4";
      $durations["5"] = "6";
      $durations["6"] = "8/3";      //quarter triplet
      $durations["7"] = "1.33333333333333333333";      //eigth triplet
      $durations["8"] = "2";
      $durations["9"] = "3";
      $durations["0"] = "1";
      $durations["-"] = "3/2";
      $durations["="] = "1/2";
      $durations["/"] = "16/3";   //triplet???
      $durations[")"] = "2/3";    //sixteenth triplet
      
      $codes[" "] = "";
      $codes["_"] = "-";
  
      //header fields
      $headers["N"] = "T:";
      $headers["C"] = "C:";
      $headers["A"] = "S:";
      $headers["T"] = "Q:";
      $headers["S"] = "I:";
      $headers["K"] = "K:";
      $headers["B"] = "M:";
      $headers["F"] = "N:";
  

sw2abc.php

    require("sw2abcdef.php");
  $abclinecount=0;
  $beats = 0;
  $beatlimit = 0;
  $ties="";
  $lasttie=false;
  
  function AddHeader($type, $data)
  {
          global $beatlimit;
          global $headers;
          global $abcheader;
          if ($data !="")
          $abcheader[$type]=$headers[$type] . substr($data,2);
          if ($type=="B")
                  {
                  $top=substr($data,2,1);
                  $bottom=substr($data,4,1);
                  $beatlimit = (16 * $top) / $bottom;
                  }
  }
  
  function AddLyrics($data)
  {
          global $abc;
          global $abclinecount;
          global $ties;
          $replace = "_";
          $data = substr($data, 3);
          $data = trim($data, " -");
          $data = preg_replace('/\s\s+/', ' ', $data);
          $data = str_replace("- ", "-", $data);
          $data = str_replace("-", "-\n", $data);
          $data = str_replace(" ", " \n", $data);
          $words=explode("\n", $data);
            $wordties=explode(",", $ties);
           foreach($wordties as $wordtie)
                  {
                  if (is_numeric($wordtie))
                          array_splice($words, $wordtie+1, 0, $replace);
                  }
          $data=implode("", $words);
          $data=str_replace(" " . $replace, $replace . " " , $data);
          $abc[$abclinecount] = "w:" . $data;
          $abclinecount++;
  }
          
  function AddTune($data)
  {
          global $abc;
          global $abclinecount;
          global $pitches;
          global $accidentals;
          global $durations;
          global $beats;
          global $codes;
          global $beatlimit;
          global $ties;
          global $lasttie;
          if ($lasttie)
                  $ties="-1,";
          else
                  $ties = "";
          $lasttie=false;
          $thisnote=0;
          $intriplet=0;
          $I=0;
          $data = $data . " "; 
          $abcline = "";
          $item = substr($data, $I, 1);
          while ($item)
                  {
                  switch ($item):
                      case " ":
                              $I++;
                      break;
                      case "M":
                      case "m":
                      case "+":
                      case "-":
                      case "*":
                              $I=$I+3;
                      break;
                      case "W":
                      case "P":
                              $I=$I+4;
                      break;
                      case "S":
                              if ($data[$I+2]=="6")
                                      {
                                          $abcline = $abcline . " | ";
                                          $beats = 0;
                                          }
                              $I=$I+4;
                      break;
                      default:
                              $space = "";
                              $lasttie = false;
                              $bar = "";
                              //echo "data". $data[$I]; exit;
                              $note = $pitches[$data[$I]];
                              //echo $note;
                              $I++;
                              $accidental = $accidentals[$data[$I]];
                              $I++;
                              $duration = $durations[$data[$I]];
                              $beats = $beats + $duration;
                                      if ((($beatlimit/2) / ($beats)) ==1)
                                      $space = " ";
                              if ($beats >= $beatlimit)
                                  {
                                  $bar = "|";
                                  $beats = 0;
                                  }
                              $I++;
                              if ($data[$I-1]=="7")  //triplets
                                      {
                                      $intriplet++;
                                      if ($intriplet==1)
                                              $note="(3" . $note;
                                      elseif ($intriplet==3)
                                              $intriplet=0;
                                      $duration=2;
                                      }
                              //need code;
                              $code = $codes[$data[$I]];
                              if ($code=="-")
                                      {
                                      $ties=$ties . $thisnote. ",";
                                      $lasttie=true;
                                      }
                              $I++;
                              if ($note != "z")
                                      $thisnote++;
                              $abcline = $abcline . $accidental . $note . $duration .$code . $space . $bar;
                      endswitch;
                      $item = substr($data, $I, 1);
                      }
          
          //echo $abcline . "
";
          $abc[$abclinecount] = $abcline;
          $abclinecount++;
  }
  
  
  function WriteHeaderLine($field, $fp)
  {
          global $abcheader;
          $temp = @$abcheader[$field];
          if ($temp != "")
              //echo $temp . "
";
             fwrite($fp, $temp . "\n");
          
  }
  
  function WriteHeader($fp)
  {
          global $abcheader;
          fwrite($fp, "X:1\n");
          WriteHeaderLine("N", $fp);
          WriteHeaderLine("C", $fp);
          WriteHeaderLine("A", $fp);
          WriteHeaderLine("F", $fp);
          WriteHeaderLine("B", $fp);
          fwrite($fp, "L:1/16\n");
          //WriteHeaderLine("T", $fp);
          //WriteHeaderLine("S", $fp);
          WriteHeaderLine("K", $fp);
  }
  
  function convertsw($ABC)
          {
          global $abclinecount;
          global $abc;
   
          //start to play with the sw
          //make array;
          $sw = split("
", $ABC);
          //start
          foreach ($sw as $swline)
                      {
                  $temp = substr($swline,0, 1);
                      switch ($temp):
                  case "N":
                  case "C":
                  case "A":
                  case "T":
                  case "S":
                  case "K":
                  case "B":
                  case "F":
                  AddHeader($temp, $swline);
                  break;
                  case "H":
                  break;
                  case "M":
                  AddTune($swline);
                  break;
                  case "L":
                  AddLyrics($swline);
                  break;
                      endswitch;
                      }
          $fp = fopen("output.abc", "w");
          WriteHeader($fp);
          for ($I=0; $I<$abclinecount; $I++)
                  {
                  fwrite($fp,$abc[$I]."\n");
                  }
          fclose($fp);
          }
  ?> 




	

Subject: RE: Tech: Digital Tradition Programmer Needed
From: dick greenhaus
Date: 20 Oct 07 - 12:05 PM

Thanx Jon. I'll need to che on that for a while.