Tech: How can I get keywords for each lyric?

Subject: Tech: How can I get keywords for each lyric?
From: everfool
Date: 05 Apr 07 - 09:59 AM

Dear All:
Hi, will anyone do me a favor and tell me how could I get the keyword list for each song? I have downloaded the latest version but I can't found how to do that and the index file seems to be non-text.
Thank you very much!
   best
wishes!

Subject: RE: Tech: How can I get keywords for each lyric?
From: MMario
Date: 05 Apr 07 - 10:21 AM

For the DT? I think it's a full text search.

Subject: RE: Tech: How can I get keywords for each lyric?
From: everfool
Date: 05 Apr 07 - 12:28 PM

Yes, for the DT, it provides the search based on keywords. I just wonder if I can do the reverse: get the keywords for a given lyric?
Thank you!

Subject: RE: Tech: How can I get keywords for each lyric?
From: Mick Pearce (MCP)
Date: 05 Apr 07 - 12:44 PM

I can provide you with a listing if you like.

Mick

Subject: RE: Tech: How can I get keywords for each lyric?
From: MMario
Date: 05 Apr 07 - 12:45 PM

in the dos version they are listed at the bottom of the page. Don't know if they display in the windows or mac version.

yes, indeed they do. If you [ull up the lyric , you get the split box, with the music above, the lyrics below. In the bottom frame of the lower box you will see the keywords.

Subject: RE: Tech: How can I get keywords for each lyric?
From: Joe Offer
Date: 05 Apr 07 - 12:50 PM

Hi, Everfool -

The Windows version of the Digital Tradition does not display keywords, although it does provide a keyword search. The online version and the DOS version both have keywords listed at the bottom of each song.

I don't think the keyword search utility is working on the online version.

Take a look at this song (click). At the bottom of the lyrics, you'll see the keywords:

@Irish @parting @emigration

You can find all the emigration songs by searching for @emigration

-Joe Offer-

Subject: RE: Tech: How can I get keywords for each lyric?
From: MMario
Date: 05 Apr 07 - 12:55 PM

joe - I think they do display at least for some of the songs - see my post above.

Subject: RE: Tech: How can I get keywords for each lyric?
From: JohnInKansas
Date: 05 Apr 07 - 02:09 PM

The files generally are non-text, and extracting the text only to look for a complete listing of keywords associated with each song is something of a challenge.

If you can download the DOS version and unzip it somewhere separate from where you keep the Windows version that I assume you'd rather use, the file "KEYS" contains a plain text list of all the keywords that are used; but won't tell you which keywords are used for each song. You can copy "KEYS" somewhere and change the name to KEYS.TXT, and it will open in notepad, but will be somewhat "unoriented." If you right-click on it (with or without the change to .txt) and "send to" Word, everything should line up legibly, and in Word you can do a few global replace operations to replace the multiple spaces with tabs (^t) or paragraph breaks (^p) if you want.

The batch file, KEYWORD.BAT, is intended to let you print a list of the keywords, but probably won't run in recent windows "DOS Windows" since it uses a command that was dropped from the Command set. The simplest way to use the DOS version probably would be to just boot to an older DOS, but if your hard drive is formatted NTFS DOS can't read anything from it, so you'd have to make a separate partition (FAT or FAT32 format) to put the DOS DIGITRAD in, or run it from a separate drive (CD?) to do it that way.

In the DOS version, the file named "TITLES" can be opened in the same way as the KEYS file, and is legible in Word, giving a list of all the tune titles. A "vertical bar" that's used as a separator can be replaced with a paragraph break by putting "^124" (without the quotes - the ANSI value of the character is 124) in the Word "Find" box and "^p" (without the quotes) in the "Replace With" box and clicking "Replace All" if it makes it easier to read/use. Again, there is no association of keywords with the individual titles.

If you open (from the DOS version) the file named "Z02.ASK" in Word and replace

"^012" with "^p" - Replace All
"^014" with "^p" – Replace All
"^027^002" with ^p^p – Replace All (allow time, >8,500 replacements)

You'll get the full text with each song separated by double paragraph breaks. The keywords for each song will be at the end of the song, preceded by an "@" for each keyword. There will be extraneous "unprintable" characters, and the document will be around 3,418 pages long, so if you're short on memory the last "replace all" especially may take a very long time, or may crash (lock up) your Word. It worked fine with my Word 2002, with 1GB RAM.

Some of the remaining "unprintables" are outside the ANSI charvalue range, and can't be easily "replaced out" in Word, but you can of course do a bit of manual cleanup.

There appears to be an "end section" in this file where the titles and keywords are associated. If you do a Find "HARDTAC" and do "Find Next" about 5 times, you'll get to around page 3343 (of 3418 the way my Word repaginated) where this section appears, and it goes to the end of the document. I haven't cracked the unprintables here, which would be how you'd separate the entries to get a "readable" list, but it looks like it may be possible.(?).

John

Subject: RE: Tech: How can I get keywords for each lyric?
From: MMario
Date: 05 Apr 07 - 02:35 PM

donkeyworks also displays the keywords associated with lyrics.

Subject: RE: Tech: How can I get keywords for each lyric?
From: Joe Offer
Date: 05 Apr 07 - 02:52 PM

MMario, is Jim Lawton's DonkeyWork Digital Tradition database engine still available? It worked very well, and seemed to be more stable than Windows DT (although DonkeyWorks didn't play the tunes).
-Joe-

Subject: RE: Tech: How can I get keywords for each lyric?
From: MMario
Date: 05 Apr 07 - 03:00 PM

I have a version 1.20 that works with the current database - not suyre where I downloaded it from.

I did have to tweak the ini file to point to the new database.

Subject: RE: Tech: How can I get keywords for each lyric?
From: Mick Pearce (MCP)
Date: 05 Apr 07 - 07:21 PM

I've got the Donkeywork 1.20 installer also (it's only about 300K if you want a copy Joe). The originating website (www.tabbytail.co.uk) is no longer active.

Mick

Subject: RE: Tech: How can I get keywords for each lyric?
From: JohnInKansas
Date: 06 Apr 07 - 03:03 AM

You want a list of all the tunes, with the keywords for each tune listed with the tune name. Here's one way of doing it, using the DOS 2002 Digitrad.

This is long, so follow carefully. The principles for each step are simple, but the execution can be a bit tedious.

I'll repeat the key instructions from the previous post.

1. Open (from the DOS 2002 Digitrad version) the file named "Z02.ASK" in Word.

2. Save from Word as a .doc using a new filename. (I used "Z02.doc")

3. Use Edit – Find and Replace (shortcut Alt-e, e)

In "Find What," type "^012" and in "Replace With," type "^p" [without the quotes, both places]

Note: ^012 means "the character whose ANSI value is 12" and ^p means a paragraph break.

Click "Replace All"

4. I recommend that you save the file at this point, as some computers may have difficulty with the rather large "Replace All" that follows.

5. Reopen Edit | Find and Replace (Alt-e, e)

In "Find What," type "^027^002" and in "Replace With," type "^p^p" [without quotes, both places]

Click "Replace All"

5a. If the last "Replace All" fails to complete, you can reopen the document you saved at step 4, select (highlight) half of the document, and repeat step 5. Then select the other half and repeat step 5 again. My Word 2002, with 1GB RAM had no trouble with it as a one-step replace, but if you have limited memory and/or temp space you might need to do it in nibbles instead of in gulps.

6. When step 5 is successfully completed, you'll get the full text with each song separated by double paragraph breaks. The keywords for each song will be at the end of each song, preceded by an "@" for each keyword. There will be extraneous "unprintable" characters, and the document will be around 3,418 pages long

Some of the remaining "unprintables" are outside the ANSI charvalue range, and can't be easily "replaced out" in Word, but you can of course do a bit of manual cleanup.

7. I suggest that you save the document again at this point. If you're nervous (which you should be), you can change the filename (Z02a.doc?).

8. Use Edit | Find (Ctl-F) with "HARDTAC" in the "Find What" box and click "Find Next." Repeat the click on "Find Next" (about 5 times total) until you get to approximately page 3,343 (of 3,418 the way my Word repaginated).

9. At the last "Find Next" you should end with the word "HARDTAC" highlighted. Hit Esc or click "Close" or click the "X" to close the Find box. Hit the "left arrow" once to move the insertion place to just in front of the word "HARDTAC" and once more to leave one character to the left of the "H" (or just click one character to the left of the "H").

10. Hold down both the Shift and Control keys, and hit the "Home" key (Ctl-Shift-Home) to select everything back to the start of the document, and then hit "Delete."

11. You now have a document consisting only of the "tail end" of the original. I suggest you save now, with a new filename (Z02b.doc?).

12. The remaining document that you just saved has each "DOS Song Name" with the title and "keywords" for each song immediately adjacent; but also contains a number of "unprintable" characters and has no line breaks to make it readable.

13. Reopen Edit | Find and Replace (Alt-e, e) and type "^031" in the "Find What" box, and "^p" in the "Replace With" box [without the quotes, both places]

Click "Replace All"

14. You should now have a document of about 160 pages, with the DOS Tune name followed by the title and then by any keywords associated with the tune, each in a separate paragraph. Each paragraph still begins with a few "unprintable" characters.

Since this may be usable, I suggest you save one more time, with or without changing the filename (I used Z02c.doc)

15 Reopen Edit | Find and Replace (Alt-E, E) and put a single space " " in the "Find What" box, and "^t" in the "Replace With." (^t means a tab)

Click "Replace All"

16. This last step will totally screw up your document, since it replaces every space with a tab. Don't worry about it.

17. Use Ctl-A to select the entire document and then key Alt-a, v, x. (Or select Table, Convert, Text to Table) Make sure that the "Separate at Tabs" button is selected, and then hit Enter.

NOTE: This is a massive conversion. My computer handled it okay, but if you have lesser "resources" you may need to select part of the document, do the step 17 conversion on that part, and then select the remainder of the document and do step 17 again on that part, or select even smaller portions and make multiple conversions by repeating the step for each chunk.

18. At this point you will have a very large table, or several smaller ones, probably with columns running off the right of the page. Click anywhere in the leftmost column, and key Alt-a, c, c (Or use Table | Select | Column). This should select (highlight) the first column of the table. Key Alt-a, d, c (Or click Table | Delete, Column) and the first column, where all the unprintable characters are should disappear.

NOTE: If you needed to break the conversion to table(s) into smaller chunks, repeat step 17 for each table. It is STRONGLY RECOMMENDED that you save the document once the left column is deleted for all your tables (or even after each deletion).

NOTE: The next step – (19.) conversion from table to text requires immense computer resources and this step may fail. As a practical matter, it is not recommended that you try to convert a table longer than about 20 pages (or less) at a time. Key F5, type a page number, click "Go To" and then "Alt-a, t" (Table | Split Table) to split the table into smaller ones. Save when done splitting it up. Do step 19 separately for each table.

19. With your cursor anywhere in a table, select the entire table (Table | Select | Table, or Alt-a, c, t). Click Table | Convert | Table to Text (Alt-A, v, b) and again check to make sure the "Separate at tabs" is still selected.

Hit enter.

Repeat this step for each table in your document.

Save your work. (Recommended: save after each table conversion is completed.)

20. Reopen Edit | Find and Replace (Alt-e, e). Put "^t" in the "Find What" box and a single space " " in the "Replace With" box. Click "Replace All"

21. If all went well, you now have a document with the DOS name of each tune, followed by the text tune name, followed by any keywords for the tune, with each tune in a separate paragraph.

22. I suggest you save with a new filename (Tunes and Keywords.doc or similar).

Your final document, per Word Tools | Word Count should have about 8,982 paragraphs. Due to a quirk in Word's global replace function, it occasionally drops a "paragraph marker" that's "strange" and doesn't get found, so this isn't necessarily an accurate count of the number of tunes indexed. In 10 point type, my conversion is 159 pages, 1,295 KB

You may want to do some additional cleanup and formatting, but that's optional to suit your purposes. There are a couple of places where "strange unprintables" resulted in the DOS name being preceded by a tab. You can remove these "errors" replacing "^p^t" with "^p" – Replace All.

John

Subject: RE: Tech: How can I get keywords for each lyric?
From: everfool
Date: 06 Apr 07 - 07:39 AM

Done!
Thank you all ! Especially John.
Really appreciate your help!!